Month: October 2013

Mail Cluster Degraded Performance

At approximated 17:15 this evening, one of the 4 clustered enterprise class filers that handles email storage started to show unexplained high CPU utilization leading to slow I/O performance.  This may have resulted in slow interactive performance or even some timed-out connections for POP3 or IMAP for approximately 25% of our users. 15 minutes later we issued a partner takeover, and a few minutes after that, a partner giveback to affect as clean a reboot of the affected system.  At this time things seem to be behaving normally but we are keeping a close eye on these systems.

-Kelsey and Nathan

Update:  This issue is ongoing and we are working to resolve.  We have no ETR at this time.  -Kelsey

Update:  We believe we have identified the source of the problem and resolved it for the time being.  It looks like the root cause may actually be a software defect of sorts in the IMAP server and not the filer.  -Kelsey

Major Core Power Work

Update (3:00AM):  This work has been successfully completed.

Tomorrow night, Tuesday October 15th, beginning at 11:00 PM, we will be performing major power work within a core datacenter in Northern California. While we expect no customer impact during this procedure, due to the length and nature of this operation we are starting earlier than normal but plan on taking on the potentially intrusive work after  midnight.

– Robbie and Nathan

Emergency Equipment Maintenance

We will be performing emergency maintenance shortly on equipment serving Legacy DSL customers in the Los Angeles area. Affected customers may see up to 10 minutes of downtime while the work is performed.

-Tim

Update: Maintenance work has been completed and all affected customers should be back online. We apologize for the downtime.

Updates to Our Recursive DNS Servers

On the morning of  Tuesday the 22nd, we will block access to our recursive DNS servers from off of our network.  This will help reduce the use of our DNS servers as part of the ongoing DNS Amplification Attacks as well as protect their quality of service for all of our customers.  This will, however, cause issues for any roaming device that has our DNS servers hard coded.  These devices should be configured to use server assigned DNS settings instead.  For more information please see our forums.

-Kelsey and William

 

Non-Intrusive Backbone & Voice Network Maintenance

Update: 4:08A PDT – This maintenance is complete.

Tonight, beginning at 11:59PM PDT, I will be performing maintenance on backbone equipment in San Jose, Palo Alto, Santa Rosa and San Francisco as well as maintenance to our voice network in Santa Rosa. This is the completion of the maintenance from 10/9/2013. No impact is anticipated to customer internet or voice traffic during this maintenance.

– Tim J.

Non-Intrusive Backbone Maintenance

Tonight, beginning at 11:59PM PDT, I will be performing maintenance on backbone equipment in San Jose, Palo Alto, Santa Rosa and San Francisco. No impact is anticipated to customer traffic during this maintenance.

– Tim J.

 

— UPDATE: Maintenance partially completed, remainder will be re-scheduled for another night.

Emergency Router Maintenance

Update (12:50AM) : This maintenance is now complete.

 

Tonight, beginning at 11:59 PM, I will be performing emergency maintenance to replaced a failed module in a core northern California router for our telecom platform. I do not expect any customer downtime as a result of  this procedure, although there may be a brief moment of routing instability as I bring the replaced module back online.

– Robbie

Server Maintenance

Update: All upgrades complete.

System Operations will be applying software upgrades between 12 and 2am on Thursday, October 3rd, 2013. The following Sonic.net systems may be unreachable for a short period of time:

– Game servers
– Mailing lists
– Mirrors
– Speed Test

— Joe and the SOC