Category: Network

Sonic Data Center Core Maintenance – Night 2 of 2

Update(1:50AM) Maintenance ran long due to unforeseen hardware issues, but all work has been been successfully completed.

Wednesday, January 8, starting at 11:59pm, we will be performing maintenance on core equipment in our Santa Rosa HQ Data Center. Minimal customer impact is expected, with brief periods of routing instability towards some Sonic services, such as web-hosting, email and Member Tools. This maintenance window is 4 hours.

-Network Engineering

Sonic Data Center Core Maintenance – Night 1 of 2

Update(1:20AM) All work for tonight has successfully concluded.

Tuesday, January 7, starting at 11:59pm, we will be performing maintenance on core equipment in our Santa Rosa HQ Data Center. Minimal customer impact is expected, with brief periods of routing instability towards some Sonic services, such as web-hosting, email and Member Tools. This maintenance window is 4 hours.

-Network Engineering

Lata 1 Northern California Legacy DSL Outage

Update: After a technician was dispatched for hardware replacement, this issue should have been resolved as of 3:30PM today.

 

Today just after 12:00pm, a piece of network infrastructure serving a subset of legacy DSL customers in northern California appears to have failed. We are currently identifying the exact source of the failure.  Update will be provided when we have more info and/or an ETR.

-Network Engineering

Non Intrusive Core Network Maintenance

Update(1:47am): Due to a configuration error, a subset of customers experienced 10-15 minutes of down time during this maintenance window. At this time, all services have been restored, and maintenance is now complete.

Monday, Jan 13, starting at 11:59pm, we will be performing maintenance on public peering interfaces at the edge of our network. Customers may experience a small period of network instability. Maintenance window is four hours.

-Network Engineering

Sonoma County Business Park Fiber Intrusive Maintenance

Update: 1:00AM – Maintenance successfully completed.  Customer downtime should have been limited to approximately 30 minutes.

Monday, December 16, starting at 11:59pm, we will be performing intrusive maintenance on equipment serving a subset of Business Park Fiber customers in Corporate Center/Northpoint business parks. Affected customers are expected to experience approximately 30 minutes of downtime. This maintenance window is 4 hours.

-Network Engineering

Northern California Fiber DIA-SSE Maintenance

Update(12:45AM): Maintenance has been successfully completed.

Next Monday (11/25) at 11:59PM, we will be performing upgrades on equipment serving all Sonic enterprise Fiber DIA-SSE customers in Northern California. Connectivity will be impacted during the course of this work. Our maintenance window will commence until 4:00AM, but we expect actual customer downtime of less than 60 minutes. An update will be provided on the completion of our maintenance.

-Network Engineering

Northpoint/Corporate Center – Business Park Fiber Maintenance

Update(12:30AM): Maintenance has been successfully completed.

Tomorrow, (11/5/19) at 11:59PM, we will be performing upgrades on equipment serving enterprise fiber customers in the the Northpoint/Corporate Center business park. Connectivity will be impacted for all customers during the course of this work. Our maintenance window will commence until 4:00AM, but we expect actual customer downtime of fewer than 30 minutes. An update will be provided on the completion of our work.

-Network Engineering

Sacramento Area – Fusion and legacy DSL outage

Update(2:11pm): The second replacement card has resolved the issue. All service restored.

Update(1:55pm): The replacement card has failed to boot, so another replacement was located and is being installed. New ETR of 3:00pm

Update(12:46pm): The issue has been tracked to a failed 100G card in our provider’s network. Dispatch for replacement has been made and we have an ETR of 1:30pm

Update(10:53am): Further investigation has narrowed the problem down to an equipment failure in our transport provider’s network. We are working with them to restore service as quickly as possible but don’t have an ETR as of yet. We would also like to take this opportunity to update the scope of the outage, which we have determined affected Fusion and FlexLink services in the Sacramento area, as well as Legacy DSL subscribers in the Sacramento and Stockton areas.

At 8:05AM this morning, a hardware serving a backbone link to Sacramento failed.  We are currently assessing the situation, but presently believe connectivity is impacted to Sonic Fusion and legacy DSL customers in the Sacramento area.  We will update with an ETR when it is available.

-Network Engineering

Northern California Network Edge Maintenance

Update(1:45am): Maintenance is complete

Tonight, October 24, starting at 11:59pm, we will be performing maintenance on our public peering interconnections. Customers may experience a brief period of routing instability towards destinations outside of the Sonic network.

-Network Engineering

See https://en.wikipedia.org/wiki/Peering#Public_peering for more information on public peering