Tonight, November 25, starting at 11:59pm, we will be performing maintenance on equipment terminating customer IPv6 tunnels. Please note this maintenance will only affect connectivity for customers who have explicitly opted in to our IPv6 tunnel services. Expected downtime for v6 tunnels is 10 minutes.
Update: Maintenance complete. Due to an issue with one of our upstream providers, v6 downtime lasted approximately 50 minutes. All service was restored by 1:05AM.
-Tomoc
Update – 11:58PM PST – This maintenance is now complete.
Tonight starting at 11:35PM PST, we will be performing a software upgrade of core voice equipment in Santa Rosa. There is no service impact expected during this upgrade, and this is expected to last 15-20 minutes.
Tonight, Wednesday, November 20th, starting at 11:59pm, we will be performing non-intrusive network maintenance in our Northern California core network.
Update: Due to a configuration error, customers may have experienced issues reaching destinations outside of the Sonic.net network. All reachability issues were resolved within 10 minutes.
-Tomoc
Tonight, November 15th, at 11pm, Operations will be taking one of our internal database servers offline to clear a hardware error. Work is expected to take no more than 30 minutes.
Update: All work is complete.
— Joe and the SOC
We are currently investigating an outage affecting all Fusion and FlexLink customers in the Mendocino area. Further details will be provided as soon as possible as we work towards resolution of the issue.
-Tim
Update: Service should be restored to all customers as of 8:08AM. We are investigating the cause of the outage further.
Tonight, October 29, starting at 11:59pm, we will be performing intrusive maintenance serving FRATM customers in Northern California. Expected downtime is less than 10 minutes.
Update: Maintenance complete
– Tomoc and Robbie
We are currently investigating an outage affecting legacy DSL customers in the Chico area. We are working on locating the cause and resolving the issue as soon as possible.
Update: All affected customers should have stabilized, we are working with AT&T to determine the cause of the issue.
-Tim
This morning around 9:35 am both of our redundant servers that serve several of our websites including webmail, forums, wiki and our blog became unresponsive and caused the sites to become unavailable. The sites are back up now and we are investigating what caused this outage.
-William and the SOC
The NOC hotline for business customer support is presently experiencing
intermittent call completion issues. We are working to diagnose and resolve this
problem, but in the mean time please contact support at 707-547-3400 and
request to be transferred to the NOC if necessary. An update will be
posted when this service interruption has been restored.
NOC
Update: The call completion issue with the NOC hotline should be resolved now. The issue was with one of our upstream providers.
At approximated 17:15 this evening, one of the 4 clustered enterprise class filers that handles email storage started to show unexplained high CPU utilization leading to slow I/O performance. This may have resulted in slow interactive performance or even some timed-out connections for POP3 or IMAP for approximately 25% of our users. 15 minutes later we issued a partner takeover, and a few minutes after that, a partner giveback to affect as clean a reboot of the affected system. At this time things seem to be behaving normally but we are keeping a close eye on these systems.
-Kelsey and Nathan
Update: This issue is ongoing and we are working to resolve. We have no ETR at this time. -Kelsey
Update: We believe we have identified the source of the problem and resolved it for the time being. It looks like the root cause may actually be a software defect of sorts in the IMAP server and not the filer. -Kelsey