Month: August 2022

Berkeley – Fiber Intrusive Maintenance

Update 9/2/22 2:20AM: Tonight’s maintenance is successfully completed.

Friday (9/2/22), starting at 1:00AM, a subset of customers in the Berkeley area will experience a brief service interruption while we perform some maintenance. The expected down time is about 30 minutes while the equipment upgrades. The maintenance window will be 2 hours.

– Network Engineering

Sebastopol – Emergency Fiber Intrusive Maintenance

Update(12:30) Maintenance is now complete.

 

Wednesday night (8/31/22), starting at 11:59PM, a small subset of customers in the Sebastopol area will experience a brief service interruption while we perform maintenance on hardware serving their connections. The expected down time is about 10 minutes. The maintenance window will be 1 hours.

– Network Engineering

Albany/Berkeley – Fiber Intrusive Maintenance

Update(1:19am) This maintenance is now complete.

 

Friday morning (9/2/22), starting at 12:00AM, a subset of fiber and Business fiber customers in the Albany and Berkeley areas will experience a brief service interruption while we perform maintenance on hardware serving their connections. The expected downtime is about 45 minutes. The maintenance window will be 4 hours.

– Network Engineering

Northern California Service Disruption

Update 8/28/2022 3:16 AM: The failed hardware has been replaced and full redundancy has been restored.

Today 8/27/2022, around 6:20 PM Sonic experienced a hardware failure on one of our main edge routers. The symptoms indicated a restart of the affect line card would resolve the issue, so we did so. Upon booting, the previously failed interfaces began discarding all outbound traffic, causing routing failures towards some internet destinations from 7:00 PM to 7:15 PM. We have taken the affected piece of hardware out of service and our redundant edge router is taking over the traffic. As of now, we appear to be stable. We are working on dispatching out to replace the failed hardware to restore full redundancy.

-Network Engineering

Mail service outage

UPDATE: 10:59pm Mail access restored.

 

During maintenance this evening starting at roughly 10:40 pm, our IMAP/POP3 cluster suffered a service impacting failure. We are working to restore mail access as soon as possible.

– SOC

Albany/Berkeley – Fiber Intrusive Maintenance

Update(3:05) Maintenance is now complete.

 

Friday morning (8/26/22), starting at 1:00AM, a subset of fiber customers in the Albany and Berkeley area will experience a brief service interruption while we perform maintenance on hardware serving their connections. The expected downtime is about 45 minutes. The maintenance window will be 4 hours.

– Network Engineering

Power event in Santa Rosa Data Center

At 3:42 PM this afternoon one of the UPSes that supplies power to our Santa Rosa data center had it’s load disconnected during routine maintenance.  Power was restored to approximately 3 minutes later at 3:45 PM.  The affected UPS handles both our own internal loads as well as our colocated customers.  The overwhelming majority of our systems are supplied with redundant power fed from two independent power systems and no public services experienced anything more than a transient failure.  More information will be forthcoming once our internal investigation of the procedural failure has completed.  – Kelsey

Southern California Core Maintenance

Beginning at 11:59PM on Wednesday, August 24th, we will be carrying-out network upgrades on a portion of our core network in Southern California. Sonic enterprise SSE customers in the region will experience a 1 or 2 hour downtime as we move equipment that is essential to that particular service. All other services may notice some routing changes but should not experience any downtime.

 

Update(9:15PM): A latent misconfiguration causing traffic loss for many subscribers in the Southern California area was exposed around 8PM when draining transit traffic in preparation for this operation. Due to the nature of the traffic loss our network monitoring did not catch the issue immediately. We were able to restore all traffic by 9:05PM and apologize for the inconvenience. We will be taking steps to protect against this type of failure in the future.

SPF record added to domains with Sonic DNS & Email hosting

Today on August 22, 2022, we added a default SPF (Sender Policy Framework) record to every customer domain name that has Sonic DNS hosting and email hosted with Sonic.

On August 15th, we emailed everyone who would be impacted by this change. The email subject was “Important information about your Sonic domain email hosting” and included a list of domain names that would be updated.

If you did not receive the email or do not have domain email and DNS hosting with Sonic, this notification does not apply to you.

If you received the above notification email and did not opt out or add an SPF record before today, the following SPF record was added to your domain:

"v=spf1 include:mail.sonic.net ~all"

For more information, please see our help document SPF and Sonic. To learn more about SPF, see Sender Policy Framework on Wikipedia.

We made this change to fight spam & phishing and to help ensure reliable email delivery for our domain email customers. SPF tells other email providers that Sonic is authorized to send email on behalf of these hosted domain names.