Today at approximately 2:05 PM a set of our Sonic SSE Fiber customers began experiencing traffic throughput issues and packet loss. We identified the cause of this disturbance as a byproduct of an unexpected network build provided by our fiber vendor. We promptly mitigated the issue and we have taken preventative measures to avoid this disturbance in the future.
UPDATE: upgrade complete.
Beginning at 11:59pm our Operations team will be upgrading the switches in our virtualization cluster. This may lead brief interruptions of tools and services.
-SOC and NOC
Maintenance was completed as of 12:30AM.
Tonight at midnight, we will be performing a second attempt at the planned migration of hardware serving vpn.sonic.net in our Santa Rosa datacenter. Expected downtime for VPN access is expected to be 30 minutes or less.
Update: As of 9:35pm, the repair work is complete. The damage was greater than initially anticipated, but all customers are now back online. Affected customers may have to reboot local equipment to restore connectivity. Thank you for your patience.
Early yesterday evening, there was a transformer fire in the Richmond District of San Francisco at 15th & Geary. Unfortunately, the fire impacted our infrastructure in the area, and some customers may have subsequently experienced limited, or no internet connectivity.
We have field engineers on site working to replace and repair the damaged portion of our network as quickly as possible. We believe the damage is limited in scope, and the current estimated time of repair is 5pm, at which point all affected customers should have their service restored.
If you have any questions in the meantime, or if your service is not restored by the estimated time of repair, please give support a call at your earliest convenience. Support can be reached at 1-888-766-4233, seven days a week, from 8am to 10pm.
We appreciate your patience in addressing this issue, and hope to have you up and running soon.
Update (1:04AM): Maintenance complete.
Update: This Maintenance has been postponed to tomorrow night at midnight.
Beginning tonight at midnight, we will be performing upgrades on equipment serving Fusion Fiber customers in the Richmond District of San Francisco. Expected downtime for this operation is approximately 10 minutes.
Today, March 19th, starting at 2:00am, equipment serving a small subset of FlexLink Long Range customers in the Oakland area failed. We are dispatching to replace the affected equipment immediately. Expected time of repair 5:30am.
Update(4:45am): Affected hardware replaced and service has been restored
A portion of calls to some Sonic telephone numbers are currently experiencing an issue where the source caller will hear a loud buzz similar in sound to a fax transmission. Our customers are commonly reporting little to no audio and a dropped call most commonly due to the source caller hanging up upon hearing this noise.
We have isolated the issue to be outside of our network and thus beyond our ability to repair directly ourselves. We understand the impact it is having on our customers and all parties involved and we are currently working with our vendors around the clock to escalate this issue as quickly as possible. We do not have an exact ETR at this time, but updates will be posted as they become available.
After extensive troubleshooting to narrow down which telco carrier could have been at fault and many hours actively working with the carrier to replicate and test the issue we are pleased to announce that as of 10:10am on 4/5/2017 this has been marked resolved.
The scope of the disturbance was isolated to calls originating outside of our serving LATA (722) and destined for telephone numbers on Sonic’s local switch. In layman’s terms; long distance calls. A portion of calls that traversed a telco carrier’s switch outside of our network were routed across T1 carrier links, two of which were corrupt. The noise itself was an amplified 60hz hum induced in some DS-1 signaling equipment that connected the offending carrier’s telco switch to the ILEC local switch.
The disruption to both personal and business related telephone service had a real and tangible effect on a portion of the residential and enterprise grade customers that we are proud to serve. From both myself here in the NOC, and all of us at Sonic; We are terribly sorry for the length of time it took to get the aforementioned carrier to acknowledge the issue and to then act upon it.
– Michael Tardif
– Network Operations
Update (12:30AM): Maintenance complete.
At midnight tonight, we will be performing non service impacting maintenance on network hardware serving the Santa Rosa and Petaluma business park fiber customers. There is no expected customer downtime for this operation.
UPDATE: Maintenance concluded for tonight. The planned migration was unsuccessful so we reverted back to the original configuration. VPN access should now be fully available.
Tonight at midnight, we will be performing a planned migration of hardware serving vpn.sonic.net in our Santa Rosa datacenter. Expected downtime for VPN access is expected to be 30 minutes or less.
Beginning tonight at midnight, we will be performing non-intrusive maintenance on core routing equipment serving our telecom network in Northern California. We do not expect any customer impact.