At approximately 4PM this afternoon we received multiple reports of outbound calls failing to complete on our voice network. We traced the problem to an upstream provider and have presently routed around them.
-Sonic NOC
FRATM Maintenance
Tonight, October 29, starting at 11:59pm, we will be performing intrusive maintenance serving FRATM customers in Northern California. Expected downtime is less than 10 minutes.
Update: Maintenance complete
– Tomoc and Robbie
Legacy DSL Outage
We are currently investigating an outage affecting legacy DSL customers in the Chico area. We are working on locating the cause and resolving the issue as soon as possible.
Update: All affected customers should have stabilized, we are working with AT&T to determine the cause of the issue.
-Tim
webmail, forums, wiki outage
This morning around 9:35 am both of our redundant servers that serve several of our websites including webmail, forums, wiki and our blog became unresponsive and caused the sites to become unavailable. The sites are back up now and we are investigating what caused this outage.
-William and the SOC
NOC hotline unreachable
The NOC hotline for business customer support is presently experiencing
intermittent call completion issues. We are working to diagnose and resolve this
problem, but in the mean time please contact support at 707-547-3400 and
request to be transferred to the NOC if necessary. An update will be
posted when this service interruption has been restored.
NOC
Update: The call completion issue with the NOC hotline should be resolved now. The issue was with one of our upstream providers.
Fusion/Flexlink Network Maintenance
Update (1:10AM) : This maintenance is now complete.
Tonight, October 21st, beginning at 11:59 PM, I will be performing maintenance on equipment that serves Fusion and Flexlink customers in the Santa Cruz, Aptos, and Watsonville areas.
– Robbie
Non-intrusive Voice Network Maintenance
Update: 12:41AM PDT 10/22/13 – This maintenance is now complete.
Tonight starting at 11:59 PM PDT we will begin working on our voice infrastructure in Santa Rosa. This is expected to last 2-3 hours. Hosted PBX customers may have intermittent access to commportal during the upgrade, but no other impact is anticipated.
-Tim J.
Fusion/Flexlink Voice Issues
We are currently experiencing issues with in and outbound calling through our voice platform. Our engineers are working on identifying and resolving the problem ASAP.
– Noc
Update: All call issues should be resolved as of around 1:50PM. We are presently working on a root cause analysis and will provide more information as it becomes available.
Mail Cluster Degraded Performance
At approximated 17:15 this evening, one of the 4 clustered enterprise class filers that handles email storage started to show unexplained high CPU utilization leading to slow I/O performance. Â This may have resulted in slow interactive performance or even some timed-out connections for POP3 or IMAP for approximately 25% of our users. 15 minutes later we issued a partner takeover, and a few minutes after that, a partner giveback to affect as clean a reboot of the affected system. Â At this time things seem to be behaving normally but we are keeping a close eye on these systems.
-Kelsey and Nathan
Update: This issue is ongoing and we are working to resolve. We have no ETR at this time. -Kelsey
Update: We believe we have identified the source of the problem and resolved it for the time being. It looks like the root cause may actually be a software defect of sorts in the IMAP server and not the filer. -Kelsey
Major Core Power Work
Update (3:00AM): Â This work has been successfully completed.
Tomorrow night, Tuesday October 15th, beginning at 11:00 PM, we will be performing major power work within a core datacenter in Northern California. While we expect no customer impact during this procedure, due to the length and nature of this operation we are starting earlier than normal but plan on taking on the potentially intrusive work after  midnight.
– Robbie and Nathan