Author: Jerriel Mcdonald

Independence Day

Hello Sonic Customers!

Whether you’re celebrating with with friends and family or taking this time to reflect on our country, we here at Sonic would like to wish you a Happy Independence Day. We will have modified hours of operation for customer care, and will be open from 8am-5pm 7/4/21 and will resume normal business hours on 7/5/21. Our 24×7 network operations team will remain on call for our Enterprise, dedicated circuit and collocated customers.

 

-Jerriel & Sonic Support

Planned Fiber Maintance for Albany

Hello Sonic Customers and Friends!

 

We will be performing a planned Fiber Maintenance on Thursday evening 4/15 at 10 pm and should have customers back up by 6 am on Friday morning 4/16.  We will be replacing a Fiber cable in the area that supports the backbone of our services. During this time both Internet and Phone service will be down. An email will be sent out to those customers affected. If you still have issues after the service window is complete please try power cycling your ONT. If this fails as well we will have Staff on standby at 8am PST. Expect higher then regular hold times .

 

-Sonic

 

Holiday Hours for 12/24 -12/25

Hello my Super Sonic Friends!

 

Sonic Support will be closing early for Christmas Eve and Christmas Day. On 12/24 our operating hours for Technical Support Team will be from 8am-5pm. On 12/25 we will be closed. We will resume regular operations on 12/26 at 8am.
We all here at Sonic wish you a happy holiday!

 

 

Emergency Card Reboot

Hello,

Emergency maintenance is required  on 12/17 between 5pm and 6pm to fix a one way audio issue. We estimate that service will be impacted for up to 15 minutes. Customers impacted have been contact via email and text message. Thank you very much for your patience.

Regards,

Sonic Support

Sonic Support holiday hours for 11/26/2020

Hello Sonic Customers!

 

We at Sonic hope you have a safe and relaxing holiday! Sonic Support will have modified hours for 11/26/2020. We will have our phone, sms(text messaging) and email support will be open between the hours of 8am-5pm.

 

Happy Gobble Gobble Day!

 

-Sonic Management.

Web Call Back Assistant Temporarily Unavailable

We’re currently experiencing issues with our web based call back assistant. We have temporarily turned off this functionality while we work to restore it to proper working order, and apologize for any inconvenience this may cause. Support is still reachable at 1-855-394-0100. Thank you for your patience while we work to resolve this matter

Service Outage in San Francisco

As of 7/31/2020 at 4pm we are experiencing an outage on our AT&T resold internet services (IPBB). The area affected is in San Francisco. We do not have an ETA but we will keep everyone updated via this post. If you are calling in please know that our hold times are higher than usual due to this outage. Please be patient with our Techs on the phone as we work to resolve this outage.
Update: 8/3/2020 9:34am: It is looking like all customer’s services should be up and running again. If you are having any issues please contact Support.

[UPDATE : RFO] Known Outage in the Northern California Area.

–START RFO–

Some Sonic fiber customers experienced an internet outage on the evening of Wednesday, April 8th. Below is the Reason For Outage (RFO) write-up, an internal process document which we are sharing to provide some insight into this unprecedented event.

On April 8th, 2020 at 6:44 PM PDT, one of the large core routers in the Sonic network failed. We have redundant equipment that is configured to take over the load immediately. Unfortunately, due to the way the router failed, the failover time to our redundant router was slower than expected and took a bit over forty seconds. During this time, roughly half of the inbound, and a significant majority of outbound internet traffic was stalled. Most data traffic in Northern California was affected, though data traffic on IPBB and in Southern California was unimpacted. Voice services network-wide were also briefly impaired.

Once traffic rerouted around the failed core router, we noticed traffic levels were much lower than expected. We were missing many tens of Gbps of traffic on transit and peering sources combined. Our Network Engineering team began investigating immediately.

After routing stabilized, we began receiving reports via Sonic’s support team of some Fusion Fiber customers in an offline state. After verifying examples, we noticed that affected customer premise equipment was stuck in a DHCPDISCOVER state, unable to obtain an IP address. Rebooting the home router and ONT (Optical Network Terminal – the devices that terminate the fiber optic connection at the customer premises) did not work to obtain an IP address and restore service.

What we found was that the brief interruption in internet access caused some customer premise devices to re-send large quantities of DHCP messages which overwhelmed their associated customer aggregation routers across our network. This resulted in those aggregation routers becoming overloaded and unable to hand out new IP addresses. Customers that had an IP address before the outage and did not release the address during the minute-long outage window were unimpacted.

Throughout the evening, the Network Engineering team worked to isolate the cause and mitigate the issue. We were concerned that any heavy-handed approaches would cause a larger outage for the other customers that were not affected. We found a potential fix for this by restarting an internal process on the aggregation routers to clear out DHCP queues that were full due to the flood of requests. However, while this resolved the issue for customers in some regions, it wasn’t enough for some heavily-loaded devices.

To fully resolve the issue the Network Engineering team then deployed a configuration change to severely bottleneck the inbound DHCPDISCOVER messages. That slowed the flood enough for even the most populated aggregation routers to slowly start handing out IP addresses to customer home routers without being overwhelmed. The last few users were brought back up shortly after midnight. Coincidentally, this is also around the time when we were able to replace the trigger for all of this – the core router mentioned earlier.

We have since reached out to the vendor we use for the customer premise equipment affected in this outage. Their devices detect internet outages within 30-45 seconds and will then re-request DHCP. This is close to the 40 seconds that it took for the aforementioned failover to take place and is the likely cause why so many devices in our network started to re-DHCP and thus triggered the congestion that prevented customer routers from coming back online.

Moving forward, we are now finalizing fixes to our aggregation routers to reduce the impact of similar events.  We’re also investigating the reason for the long core router fail-over time to reduce the impact of any similar issues. Finally, we are having an ongoing post-mortem discussion with our vendor architects and engineers on this incident and will be seeking ways to prevent a similar issue from happening again.

–END RFO–

 

Update: 11:59pm We believe all services have been restored. If you are still impacted or otherwise experiencing service issues please contact our technical support team tomorrow morning.

Update: 11:30pm We have begun to restore regions in a staged fashion as we carefully monitor progress.

Update: 10:54pm We continue working on resolving the issue.

Update: 10:24pm We continue to work on resolving the issue.

Update: 9:54pm We are still working on resolving the issue.

Update: 8:53pm We currently have our network operations team looking into the DHCP issues (we started around 7pm tonight) As of right now there is no new information. We will be posting updates once new information is given to us. Thank you for your patience!

As of 7:00pm we are experiencing issues with our network that is causing DHCP issues. This is causing customers to not be able to surf or access the internet. Currently we are only seeing this issue on our Sonic Fiber services. We thank you for your time as we work to resolve this.