Some Sonic customers using copper based services in the Berkeley area may have began experiencing connectivity issues around 3:40pm this afternoon. After investigating the issue, we determined the fastest path to resolution was emergency reload of the affected hardware. This was performed around 4:10 and appears to have resolved the issue. We are currently monitoring to verify it’s success.
-Network Engineering
Update (12:53AM): This maintenance is complete.
Tonight at 11:59PM, we will be performing intrusive maintenance on hardware serving Sonic customers in northeastern Santa Rosa. Customer connectivity will be impacted for the duration of this work. The maintenance window will last until 3AM, but we expect work to conclude much sooner.
-Network Engineering
Update: Maintenance was completed at 2:00AM (10/26/18).
Tonight (10/25/18) at 11:59PM we will performing intrusive maintenance on another subset of Fusion Fiber customers located in San Francisco’s Richmond District. Approximate expected customer downtime is 30 minutes. An update to this MOTD will be provided when all customer connectivity has been restored.
Around 5:30PM today, routine network configuration on equipment serving Brentwood triggered a router bug that broke customer connectivity. Our support staff quickly alerted network engineering and we were able to revert the configuration and restore service. The NOC will in investigate the exact cause of the issue and take steps too prevent it’s recurrence.
-Network Engineering
Tonight at 11:59PM we are performing maintenance on aggregation gear serving our Brentwood fiber network. Customer impact should be minimal, but users may experience a few minutes of downtime. The maintenance window is expected too complete by 1AM.
-Network Engineering
Beginning at around 1:40AM this morning we experienced a period of routing instability on the backbone network feeding our Santa Rosa headquarters. Customers collocated in our datacenter along with Sonic member services may have experienced reachability issues for around 1 hour. We are presently investigating the root causes of this outage so we can prevent any future occurrence.
This Saturday (6/9) starting at 5AM, one of the San Francisco data centers we collocate equipment, will be performing a planned power maintenance. Although we have been assured by the colocation provider this work will be non-impactful, we have taken additional preparations to verify redundancy on all critical hardware, and will have a network engineer on site as a fail-safe. The maintenance window officially extends until 6PM, and we will provide update when all work is completed.
– Network Engineering
(Update 8:12am) – We had a Power supply fail in a piece of our equipment during the first phase of the datacenter power maintenance. We quickly swapped out the cabling to restore the unit. Legacy DSL customers would have been impacted but service should now be restored.
Update(1:03pm) Power maintenance complete.
Wednesday, starting at 11:59pm, we will be performing non-intrusive maintenance on core equipment throughout the Northern California Sonic Telecom core network. Maintenance window is 4 hours.
Update (2:10am): Maintenance is now complete, with no customer impact.
– Network Engineering
Today starting at 12:00pm, we will be performing maintenance on core routers in a series of POPs, spanning between Richmond, Vallejo, Napa, and Santa Rosa. This work should be non-intrusive to customer traffic. Estimated maintenance window is 8 hours, and an update will be provided upon completion.
-Network Engineering
Tonight beginning at midnight, a small subset of our Business Park Fiber customers may experience a brief interruption in service. The maintenance window is from 12:00am to 1:00am, with a max interruption of 5 minutes.
~Network Operations