Author: Michael Tardif

Sonic Data Center Core Maintenance – Night 2 of 2

Update(1:50AM) Maintenance ran long due to unforeseen hardware issues, but all work has been been successfully completed.

Wednesday, January 8, starting at 11:59pm, we will be performing maintenance on core equipment in our Santa Rosa HQ Data Center. Minimal customer impact is expected, with brief periods of routing instability towards some Sonic services, such as web-hosting, email and Member Tools. This maintenance window is 4 hours.

-Network Engineering

Sonic Data Center Core Maintenance – Night 1 of 2

Update(1:20AM) All work for tonight has successfully concluded.

Tuesday, January 7, starting at 11:59pm, we will be performing maintenance on core equipment in our Santa Rosa HQ Data Center. Minimal customer impact is expected, with brief periods of routing instability towards some Sonic services, such as web-hosting, email and Member Tools. This maintenance window is 4 hours.

-Network Engineering

Lata 1 Northern California Legacy DSL Outage

Update: After a technician was dispatched for hardware replacement, this issue should have been resolved as of 3:30PM today.

 

Today just after 12:00pm, a piece of network infrastructure serving a subset of legacy DSL customers in northern California appears to have failed. We are currently identifying the exact source of the failure.  Update will be provided when we have more info and/or an ETR.

-Network Engineering

Lata 1 Northern California Legacy DSL Outage

Update 12:40AM: Replacement of failed hardware has resolved the issue and all impacted customer connectivity should be restored.

Tonight just after 6:00pm, a piece of network infrastructure serving a subset of legacy DSL customers in northern California appears to have failed.  After lengthy investigation, we have identified the source of the issue and are currently working on a repair as quickly as possible. An update will provided when we have a timeline for resolution.

-NOC Engineering

Sonoma County Business Park Fiber Intrusive Maintenance

Update: 1:00AM – Maintenance successfully completed.  Customer downtime should have been limited to approximately 30 minutes.

Monday, December 16, starting at 11:59pm, we will be performing intrusive maintenance on equipment serving a subset of Business Park Fiber customers in Corporate Center/Northpoint business parks. Affected customers are expected to experience approximately 30 minutes of downtime. This maintenance window is 4 hours.

-Network Engineering

Northern California Fiber DIA-SSE Maintenance

Update(12:45AM): Maintenance has been successfully completed.

Next Monday (11/25) at 11:59PM, we will be performing upgrades on equipment serving all Sonic enterprise Fiber DIA-SSE customers in Northern California. Connectivity will be impacted during the course of this work. Our maintenance window will commence until 4:00AM, but we expect actual customer downtime of less than 60 minutes. An update will be provided on the completion of our maintenance.

-Network Engineering

Northpoint/Corporate Center – Business Park Fiber Maintenance

Update(12:30AM): Maintenance has been successfully completed.

Tomorrow, (11/5/19) at 11:59PM, we will be performing upgrades on equipment serving enterprise fiber customers in the the Northpoint/Corporate Center business park. Connectivity will be impacted for all customers during the course of this work. Our maintenance window will commence until 4:00AM, but we expect actual customer downtime of fewer than 30 minutes. An update will be provided on the completion of our work.

-Network Engineering

Sacramento Area – Fusion and legacy DSL outage

Update(2:11pm): The second replacement card has resolved the issue. All service restored.

Update(1:55pm): The replacement card has failed to boot, so another replacement was located and is being installed. New ETR of 3:00pm

Update(12:46pm): The issue has been tracked to a failed 100G card in our provider’s network. Dispatch for replacement has been made and we have an ETR of 1:30pm

Update(10:53am): Further investigation has narrowed the problem down to an equipment failure in our transport provider’s network. We are working with them to restore service as quickly as possible but don’t have an ETR as of yet. We would also like to take this opportunity to update the scope of the outage, which we have determined affected Fusion and FlexLink services in the Sacramento area, as well as Legacy DSL subscribers in the Sacramento and Stockton areas.

At 8:05AM this morning, a hardware serving a backbone link to Sacramento failed.  We are currently assessing the situation, but presently believe connectivity is impacted to Sonic Fusion and legacy DSL customers in the Sacramento area.  We will update with an ETR when it is available.

-Network Engineering

Technical Support Department Update

Update: 10/29/2019 – 12:00pm

Sonic customers with our Fusion IP Broadband service may be experiencing service interruption due to the PG&E PSPS (Public Safety Power Shutoff). Service should be restored as soon as the power is restored. If you need support the call center is open, though expect hold times as we are experiencing high call volume and are still operating with a skeleton crew. Stay safe out there.

Update: 10/28/2019 – 12:56pm

We are now taking calls as well as responding to tickets but are still operating with a skeleton crew. Please forgive any hold times you may experience. We will update when operations are back to normal or if the situation changes.

~Sonic

 

Sonic’s Technical Support team is currently operating on a skeleton crew due to our home down being under evacuation orders as a result of the spread of the Kincade fire. As a result our Technical Support department is unable to respond to incoming calls. We are handling customer issues via tickets as quickly and accurately as we can.

To create a Technical Support ticket please email support@sonic.com. We will be resuming regular operations as soon as possible.

Thank you for your patience and understanding.

~Sonic

Sunset District – Fiber Maintenance

Update(12:55am) All maintenance successfully completed.

Tomorrow (10/15/19) at 11:59PM, we will be performing upgrades on equipment serving residential and enterprise fiber customers in the Sunset District of San Francisco. Connectivity will be impacted for a subset of customers during the course of this work. Our maintenance window will commence until 4:00AM, but we expect actual customer downtime of less than 60 minutes. An update will be provided on the completion of our maintenance.

-Network Engineering