Update 22:56:37 PDT: Everything looks good and all services appear to be functioning.
Update 21:37:03 PDT: The final cohort has been enabled. We’ll continue to watch the cluster for next several hours to ensure that everything is working as intended. It will still take some time before we’ve regained expected performance.
Update 19:17:35 PDT: Approximately 5/6 of user’s access to mail spools has been restored. We’re still on target for having services fully restored in a few hours.
Update 17:39:57 PDT: Approximately 2/3 of user’s access to mail spools has been restored and while users can expect slow performance, the cluster and storage pools appear to be stable and chugging along. At this point, we expect to have services fully restored later tonight. Please note that no email was lost during this outage and our inbound servers were able to deliver mail without issue.
Update 16:19:00 PDT: The status of our email system is continuing to improve – we will post an update as our systems return to normal operation.
Update 13:00:56 PDT: In order to allow the cluster to recover we’ve taken steps to reduce the total number of user who can log in at once. This appears to be having positive results and is allowing the storage pools to catch up with the load. We still don’t have an ETA but the situation is improving.
Update: We believe the underlying issues have been been resolved. However, it will take some time for IMAP and POP services to return to normal. The fix has resulted in substantially increased IO load on our mail spool storage as all mailbox indexes have to be rebuilt. This rebuild process can take a long time to complete, especially for large mailboxes, and it will be some time before the process is complete for all active mailboxes.
Update: We’re continuing to restore services for effected users. Please note this may also affect some Fusion DSL user’s voicemail services.
Update: mailboxes for many users have been fixed and are working properly. We’re continuing to repair remaining effected users.
Unfortunately, the mail server upgrade and rollback has caused issues for our IMAP cluster and it looks like most IMAP sessions are failing. We’re working to restore services ASAP and will have more information forthcoming. -Kelsey and Grant.
Gary,
The good news is that both your issues are likely caused by this interruption, and should be fixed once server health is 100%.
Also a customer for very close to TWO decades (hard to believe)! I’ve been to the Sonic Datacenter, many years ago and saw really top-notch infrastructure and met some really wonderful dedicated staff and was impressed with the people running the show there and in subsequent years as well…
Over the years Sonic Email and other services have generally been very reliable for me and my associates. Email problems have been especially minimal and rare.
I’ve worked extensively in IT and I know how challenging it is to ensure what most persons expect, 100% server, services, network and software up-time. Although I’m currently moderately inconvenienced, others may be more critically affected. I wish Sonic and all the staff the greatest success to get everything up again, 100% restored AND with minimal impact to your reliability reputation. Unfortunately, most humans tend to remember the “bad” and not so much the “good” when it comes to life experiences…
Here’s to remembering the almost TWO decades of really great services and looking forward to the many still to come!
1:40 STILL DOWN
Not good.
No ETA?
Please advise about settings (reset?) with Apple (Pop) mail. Error message reads: “Cannot Get Mail. The mail server ‘pop.sonic.net’ is not responding. Verify that you have entered the correct account info in Mail settings.” As Webmail is still not an option for me, shall I assume the Apple program resident on my phone and home computer will “see” mail once issues have been resolved? Or is there something else I should do at that time? Thanks. First major glitch in 20 years, but unsettling, nonetheless…
Linda,
I would hang tight for now. Once the issue is resolved on our side those messages should populate on your phone. If you still have issues after that then contacting support to go over your settings would be the next step.
Sonya,
A very possible side effect. Once the health of the cluster stabilizes I would expect your login to work properly.
Can we get an update on when this will be done?… hours down during the middle of the night is one thing, but right in the middle of the day is not exactly the best planning.
Well one thing for sure – the support team now understands the scope of the email system in relation to the user base. The next upgrade will have much better planning with a system image for a fall-back position, scheduled for minimum usage times, and perhaps a staggered implementation instead of an “all at once” approach. I’ve been in your shoes and know these are hard lessons learned.
I want to second the praise said in so many other posts. Except for this hiccup, the Sonic service and support has done an outstanding job with carrier-grade up-time and availability.
Any ideas of when i will be able to receive emails? I run a small online business and rely on the internet and email systems.
Yay! My box is downloading new mail. Thanks for the continuing updates. It really eases any frustration.
Ditto Andy’s comments. I too have been with Sonic since the early 1990s and it’s rare they have a problem. S–t happens and it’s being resolved. Thanks for the detailed updates.
It would be very helpful if you posted updates hourly, thank you.
Once Webmail is back up (only system I use) am I going to be waiting yet longer for email access? I have maybe 700-800 in my ‘inbox’ and maybe 600-700 in my ‘sent’ box. Will those be immediately available once Webmail is back up?
Are people being notified that we haven’t received their emails? This is for my business so I want my customers to know I haven’t received what they are sending me.
This isn’t good — not only the outage, but — especially — the fact that you’re not able to provide an ETA for a fix after all this time.
Max,
Yes, once webmail is back up your email should be immediately available.
I’m glad that it wasn’t an outside attack that brought the system down. Sounds like you have it under control, thanks. Security is always an issue nowadays.
Sonic’s been good for me for 10 years or more now, not flawless, but what is.
I fully understand not being able to give an ETA, when you’re trying to fix something, tech or otherwise, unless you have done it several times before, you don’t know how long it’s going to take.
That said, keep working at it folks you know how we all get if email doesn’t work.
Two hours ago, I got a peek at my email, and thought maybe it was fixed. But now I’m checking it again and getting the same “Server down” message. “We’ll be back as soon as possible.” Ugh. Dark Friday. 🙁
This doesn’t seem to be a quick fix. … After decades of uninterruped good service this hiccup is huge. Thanks for the good service and now let’s get back on track.
Any updates? We would love to be able to get back to work! Thanks!
i have nor received email for 2 days–is that because of your problems. i can send email no email is downloading. this is hard for a small business. updates would be great thanks
This has been very frustrating. I need my mail like everyone else. I own a business that depends on my email. I see no reason to really complain as I have no other control except to look elsewhere in case this happens. I have been with Sonic from the beginning.
Our business email accounts are now populating! Thank you Sonic!
Add me to the “I’ve been with Sonic†since nearly the beginning (when Zapcom in Ukiah was purchased? Merged?)
Anyway- we know y’all are doing your best. Hang in there and thanks for all the great service over the years.
It feels so weird to be disconnected. Almost like not being tethered to the phone.
Just a quick word of support to the wonderful Sonic team during this difficulty — We’ve been somewhat inconvenienced, sure, and this is not great for business. Still: our firm has been delighted with Sonic’s provision of email and web services for our business over 8 years (after a long history of horrible service from others with bigger names and reputations). I have personal services with Sonic that have been flawless. Today’s experience is the first hiccough we’ve experienced with basic services — ever — so we are considering this event in the context of consistently superior service, the demonstrated social responsibility that leads Sonic to provide truly “caring” customer care and be a strong advocate for Net Neutrality, and the honor we of having a business partner we can take pride in. Thank you all.
Now past 3 PM. Last update was ~1 PM. What’s current status? % customers restored as of now? expected time for 100% restoration??
PLS UPDATE
Is there any way you could prioritize the rebuilding of everyone’s IN box ahead of rebuilding all miscellaneous other mailboxes? That way more of us might get access to the new mails sooner and be able to conduct vital business this afternoon.
Crippling our communication.
3:33pm west coast time and *still* no email. Any ETA yet?
The post I sent a minute ago did not post: 3:36pm, still no email. Any ETA yet?
3:37pm still no email. Any ETA yet?
Hello? Please update!
Okay, I’m starting to loose patience. This has turned into an all day fiasco, and there are important emails that I need to read and respond to. @#($@#($*!!
is there a way you can put an “out of service message up” so customers/people trying to connect have an idea? I can not access any of my account.
Sorry that you’re having to work through issues and know you’re doing your best to resolve them. Do need an estimate of the amount of time email will be down though. Hours, days, weeks? Don’t expect the answer to be exact as it can be difficult but do need a rough number.
Thanks.
Those of us with your lousy phone service can’t even get messages. I have important calls I’m waiting for in some emails I can’t access this is ridiculous you even know how to update your
3 hrs since the last update? At this point we’ve lost a whole day of connectivity for our small business. Trying to be patient, but this is a pretty epic outage. Please update us.! Thanks
Hey, folks, Sonic is the best; been with you forever. If you’re having problems, you’ll fix it. But an eta would be great. And I like the idea of a “temporarily out of service” message going up; people have other ways of reaching me, if they know they need to use them.
I’m getting a password changed message requiring me to put in the correct password. I did not change my password. Is this part of the crash? Will this message go away when the server problem is fixed? Rich
Your message says that you do not have an account at my address, I have lived at this address for 8 1/2 years and your monthly fee gets charged to a credit card.
I found the helpful announcement about the problem when I hit the webmail url.
That’s *fantastic* service already: any other provider will let you hang.
Then Sonic provides a forum link on the same page, and support staff post updates to the forum.
I’ve *never* heard of an service provider do something like that.
Sonic has banked a lot of goodwill with of me since April 4, 2001.
Very bad judgment to do this work on a business work day. Why was this major work not scheduled for Saturday/Sunday, when most of us do not conduct business? Those of us who transmit time sensitive documents have been more than inconvenienced.
Any ETA when service will be restored?
WTF? Timeline, please!!!
I am transferring from Yahoo Mail because they have major problems. During the day I need to be able to send and receive emails through Sonic. Is there any way that this maintenance can be done at night when there is less traffic? Thank you.
What is the status of the server issues. I still have no access to my email account.
SIX hours down?????!!! I need to read my email. I’ve been a customer since 2002. What’s going on??
This is very bad in the mandatory items I need to finish today; this is literally putting my life back by half a week. Upgrades like this should only be done on Wednesdays when no one needs to do vital paycheck day banking (Friday) and other vital things (preparing for weekend).