Author: admin

Legacy DSL Maintenance – Northern California

Update(12:15AM): This maintenance is now complete.

Beginning tonight at midnight I will be performing maintenance on equipment that serves a small portion of legacy DSL customers in northern California. Expected downtime for this operation is around 15 minutes.

– Robbie

Offices closing early tonight

Sonic will be closed Monday, 7/27/2015 from 4pm to 10pm to celebrate Sonic’s 21st birthday! We will return Tuesday to our normal working hours, and we will call back anyone who left us a message during the closure. Our 24×7 network operations team will still be on call for our Enterprise, dedicated circuit and colocation customers.

Legacy DSL Maintenance

Tonight, July 23 starting at 11:59pm, we will performing maintenance on equipment serving a small subset of legacy DSL customers in Northern California. Expected downtime is less than 20 minutes.

Update(12:38am): Maintenance complete, downtime for affected customers was less than 5 minutes.

-Tomoc

Non-Intrusive Voice Network Maintenance

Tonight beginning at 11:59PM PDT we will be performing a software upgrade of various components of our voice network in Santa Rosa. No service impact is expected from this, but users may experience sign-outs/session time-outs on voice portals during the maintenance. Estimated time is around ~1 hour.

-Brandon

 

Update: Maintenance is now complete

-Brandon

Fusion Voice Outage

This morning a bug was encountered during last nights maintenance that was causing Fusion Voice to not work.  We have made the appropriate changes to fix the issues with certain customers and all voice should be corrected as of 10:45am.  Sorry for the inconvenience.

-Brandon

Mail Spool Filer Issues (was IMAP issues)

This morning starting at 10am we began noticing a performance impacting issue on our IMAP cluster. We are currently diagnosing the problem and hope to have the system back up and running as soon as possible. This will cause issues for mail retrieval on local clients as well as webmail.

-SOC

Update: We’ve traced the problem back to high CPU utilization on one of the heads units in one of the two NFS filer clusters used to store mail spools.  At this time, it isn’t entirely clear what is causing the problem and we’re working to trace it further to the source and determine what options we to resolve the situation.  In the meantime, POP, IMAP and Webmail services may have intermittent availability problems.  No stored mail, or inbound mail, is being deferred or lost.  -Kelsey, William and the rest of the SOC.

Update: All services have been back to normal since 11:45AM.  The problem filer is still showing elevated usage but is no longer running one of the CPUs at 100% – we still do not know the root cause but suspect it may have been one or more mail server processes stuck in a byzantine failure.  We’re continuing to investigate and monitor the situation and have contacted our vendor for assistance.  -Kelsey and William

Fusion/FlexLink Intrusive Maintenance – Sebastopol

Update(3:04AM): This maintenance is now complete.

Beginning tonight at midnight I will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in the Sebastopol area. Downtown for this operation can be up to an hour.

– Robbie

T1 Outage in Sebastopol

We are currently experiencing an issue with a piece of equipment that serves T1 customers out of Sebastopol.  We are working on the issue as fast as possible to get everything back up.  We will update as things progress

-Brandon

Update:  Service has been restored

-Brandon

Non-Intrusive Voice Network Maintenance

Beginning tonight at 12am I will be conducting a code upgrade on a piece of gear in our phone system.  There should be no downtime, at most in-progress calls may experience a drop.  Maintenance should last around ~1hr

 

-Brandon

 

Maintenance is now complete

-Brandon