Category: Uncategorized

Sonic will be closed Monday, 7/27/2015 from 4pm to 10pm to celebrate Sonic’s 21st birthday! We will return Tuesday to our normal working hours, and we will call back anyone who left us a message during the closure. Our 24×7 network operations team will still be on call for our Enterprise, dedicated circuit and colocation customers.

Tonight, July 23 starting at 11:59pm, we will performing maintenance on equipment serving a small subset of legacy DSL customers in Northern California. Expected downtime is less than 20 minutes.

Update(12:38am): Maintenance complete, downtime for affected customers was less than 5 minutes.

-Tomoc

This morning starting at 10am we began noticing a performance impacting issue on our IMAP cluster. We are currently diagnosing the problem and hope to have the system back up and running as soon as possible. This will cause issues for mail retrieval on local clients as well as webmail.

-SOC

Update: We’ve traced the problem back to high CPU utilization on one of the heads units in one of the two NFS filer clusters used to store mail spools.  At this time, it isn’t entirely clear what is causing the problem and we’re working to trace it further to the source and determine what options we to resolve the situation.  In the meantime, POP, IMAP and Webmail services may have intermittent availability problems.  No stored mail, or inbound mail, is being deferred or lost.  -Kelsey, William and the rest of the SOC.

Update: All services have been back to normal since 11:45AM.  The problem filer is still showing elevated usage but is no longer running one of the CPUs at 100% – we still do not know the root cause but suspect it may have been one or more mail server processes stuck in a byzantine failure.  We’re continuing to investigate and monitor the situation and have contacted our vendor for assistance.  -Kelsey and William

Beginning tonight at 12am I will be conducting a code upgrade on a piece of gear in our phone system.  There should be no downtime, at most in-progress calls may experience a drop.  Maintenance should last around ~1hr

-Brandon

Maintenance is now complete

-Brandon

Today, July 16, at 10:38am, fiber facilities serving a subset of Fusion/FlexLink customers in the greater Los Angeles area went down. We are working to restore service as quickly as possible.

Update(11:35am): We have routed traffic around the affected circuits and were able to restore traffic to almost all subscribers. We are working to restore facilities to the final affected office as quickly as possible.

Update(10:37am): Outages yesterday were caused by multiple fiber cable cuts in Southern California. Most facilities have been replaced or repaired, but repairs on one remaining cable is still underway. We continue to work to restore service to the last affected subscribers.

Update(10:53am): ETR for final repairs is 2 hours.

-Tomoc, NOC

Tonight, July 13, starting at 11:50pm, equipment serving a small subset of legacy DSL subscribers in the greater Bay Area suffered a hardware failure. We are working to restore service as quickly as possible.

Update: Connectivity has been restored

Update(7:47am): The affected equipment has experienced a permanent failure. We are working to migrate all subscribers to new equipment.

Update(8:14am): All subscribers have been migrated to new equipment at this time.

-Tomoc

Wednesday night, starting at 11:00PM PDT we will be performing maintenance on portions of our voice network in Santa Rosa. No customer impact is expected.

-Tim J.

Tonight beginning at 11:59PM PDT we will be performing a software upgrade of various components of our voice network in Santa Rosa. No service impact is expected from this, but users may experience sign-outs/session time-outs on voice portals during the maintenance. Estimated time is around 3-6 hours.

-Tim J.

12:02AM PDT – This maintenance is now complete.

Tonight starting at 11:59PM PDT we will be performing a slight configuration change on equipment serving Fiber DIA customers (including the Airport business park) this will be a 1-2 minute outage during the maintenance.

-Tim J.

5:24PM PDT – All services are restored.

Today, July 1, starting at 4:00pm, a fiber cut interrupted service to a subset of business park subscribers. We are working to restore service as quickly as possible with a rough ETR of 5:30pm.

-Tomoc