Tonight, November 15th, at 11pm, Operations will be taking one of our internal database servers offline to clear a hardware error. Work is expected to take no more than 30 minutes.
Update: All work is complete.
— Joe and the SOC
Webmail Outage
Tonight, between approximately 7:50p and 8:20p, webmail was unavailable for half of our customers. Operations has identified a database configuration error as the root cause and is taking steps to ensure the same failure will not happen again.
— Joe and the SOC
Mail Server Upgrades
Tonight, at 12:00AM, we will migrate “mail.sonic.net” to a new cluster. Users may be unable to send mail through mail.sonic.net during the migration which is expected to take 10-15 minutes to complete. There’s no new exciting features to tout, just a fresh OS and faster modern hardware.
In other news, our MX server cluster now supports SSL encryption for inbound connections. This allows the session between the connecting server and our’s to be fully encrypted preventing messages from being available “on the wire” in clear text. Our outbound servers have supported this kind of encryption when sending mail to destinations that support it for more than a decade. At this time it looks like approximately 15-20% of inbound email is encrypted in transit.
-Kelsey
Update: Wed Nov 13 00:48:37 PST 2013, the upgrades are finished and went over as planned.
Non-intrusive Voice Network Maintenance
11/5/13 – 12:09AM PST – This maintenance is now complete.
Tonight at 11:59PM PST I will be performing maintenance on a portion of our voice network in Santa Rosa. No customer impact is expected and the maintenance should last between 15 and 30 minutes.
–Tim J.
Non-Intrusive Voice Network Maintenance
6:05PM PDT – This maintenance has been postponed.
Tonight starting at 11:59PM PDT I will be performing some non-intrusive maintenance to our voice network in Santa Rosa. No service impact is expected from this. This maintenance is expected to last 15-20 minutes.
–Tim J.
Fusion and FlexLink outage
We are currently investigating an outage affecting all Fusion and FlexLink customers in the Mendocino area. Further details will be provided as soon as possible as we work towards resolution of the issue.
-Tim
Update: Service should be restored to all customers as of 8:08AM. We are investigating the cause of the outage further.
Outbound Voice Issues
At approximately 4PM this afternoon we received multiple reports of outbound calls failing to complete on our voice network. We traced the problem to an upstream provider and have presently routed around them.
-Sonic NOC
FRATM Maintenance
Tonight, October 29, starting at 11:59pm, we will be performing intrusive maintenance serving FRATM customers in Northern California. Expected downtime is less than 10 minutes.
Update: Maintenance complete
– Tomoc and Robbie
Legacy DSL Outage
We are currently investigating an outage affecting legacy DSL customers in the Chico area. We are working on locating the cause and resolving the issue as soon as possible.
Update: All affected customers should have stabilized, we are working with AT&T to determine the cause of the issue.
-Tim
webmail, forums, wiki outage
This morning around 9:35 am both of our redundant servers that serve several of our websites including webmail, forums, wiki and our blog became unresponsive and caused the sites to become unavailable. The sites are back up now and we are investigating what caused this outage.
-William and the SOC