Sonic DNS servers experienced a brief outage this morning starting at approximately 11:40. Services were restored within fifteen minutes. During the outage web browsing would have given errors.
We are investigating the root cause of this outage and will update appropriately.
Update: The root cause was a misconfigured security policy module propagated across the DNS cluster. Technical and policy steps to prevent this issue from occurring again have been identified and will be implemented.
Thursday (11/29/18) at 11:59PM we will performing intrusive maintenance on a subset of Fusion DSL and Flexlink customers located in Davis, Oakland, Rohnert Park and Sebastopol. Approximate expected customer downtime is 60 minutes or less. The maintenance window will be 6 hours.
Update: Maintenance completed within expected parameters
– Network Engineering
Thursday (11/29/18) at 11:59PM we will performing intrusive maintenance on a subset of Fusion Fiber customers located in San Francisco’s Sunset District. Approximate expected customer downtime is 60 minutes or less. An update to this MOTD will be provided when all customer connectivity has been restored.
-Network Engineering
Update — Maintenance was completed at 6am.
(Update:1:25PM): We have restored the circuit as of about 12:02pm today.
(Update: 11:50PM): We have narrowed down the likely location of the failure and will be dispatching a technician to coordinate some further troubleshooting with our vendor. We are estimating a repair by late morning tomorrow.
(Update 8:10PM): We are still working with our vendor to locate the problem with this circuit and will update as soon as we can.
We are currently investigating a legacy DSL outage in Stockton and the surrounding area. We will update this message when we have more information.
Some Sonic customers using copper based services in the Berkeley area may have began experiencing connectivity issues around 3:40pm this afternoon. After investigating the issue, we determined the fastest path to resolution was emergency reload of the affected hardware. This was performed around 4:10 and appears to have resolved the issue. We are currently monitoring to verify it’s success.
-Network Engineering
Tonight at 11:59 PM the Sonic Member Tools (https://members.sonic.net/) will be unavailable for a brief period of time while software upgrades are installed.
The maintenance window is 1 hour, but we expect no more than 20 minutes of down time. During this time, functions such as bill pay, Graymail management, and voice tools will be unavailable.
Update: Maintenance completed at 12:15 AM with no down time.
-System Operations
Update (12:53AM): This maintenance is complete.
Tonight at 11:59PM, we will be performing intrusive maintenance on hardware serving Sonic customers in northeastern Santa Rosa. Customer connectivity will be impacted for the duration of this work. The maintenance window will last until 3AM, but we expect work to conclude much sooner.
-Network Engineering
Tonight at 11:00pm we will be rebooting the VPN server. The expected downtime should be brief and we expect to complete all maintenance within 1 hour. Thank you.
Starting at 1PM PST on Saturday, November 10th, maintenance work will be performed on the DNS cluster in San Jose. The maintenance window is four hours with no expected customer impact.
Update: As of 11:05PM we believe we have identified and mitigated a router bug causing the issue. We will troubleshoot further with our vendor to ensure the situation does not repeat.
Starting around 6:30PM this evening, a subset of business and residential copper served customers in San Francisco may be seeing short bursts of packet loss approximately every 30 minutes. We are currently working to diagnose the cause and will provide update upon resolution.
-Network Engineering