Update (12:35AM): Maintenance complete
Beginning tonight at midnight, we will be performing software upgrades on core network equipment in Mendocino, CA. Because traffic is expected to route around the affected equipment, no customer downtime is expected.
-Network Engineering
Update: Our vendor has had to reschedule yet again, this time for May 16th and 10AM.
Update: Maintenance has been rescheduled again, this time for May 1st.
Update: Maintenance has been rescheduled for April 27th.
Update: This maintenance has been cancelled and will be rescheduled at a later date.
Tomorrow, starting at 10:00, the UPSes that service our Santa Rosa data center will be undergoing scheduled preventative maintenance. This includes overhauling one of them with a complete capacitor replacement. The maintenance is fully scripted with our vendor and no interruption of power services to the datacenter are expected. -Sonic Operations and Facilities
Update (12:18AM): Maintenance complete
Beginning tonight at midnight, we will be performing software upgrades on core network equipment in Santa Rosa, CA. Because traffic is expected to route around the affected equipment, no customer downtime is expected.
-Network Engineering
Update (12:57AM): Maintenance complete
Beginning tonight at midnight, we will be performing software updates on core network equipment. Traffic is expected to route around the affected equipment, but customers may experience a brief period of routing instability as traffic shifts.
-Network Engineering
Today at approximately 2:05 PM a set of our Sonic SSE Fiber customers began experiencing traffic throughput issues and packet loss. We identified the cause of this disturbance as a byproduct of an unexpected network build provided by our fiber vendor. We promptly mitigated the issue and we have taken preventative measures to avoid this disturbance in the future.
UPDATE: upgrade complete.
Beginning at 11:59pm our Operations team will be upgrading the switches in our virtualization cluster. This may lead brief interruptions of tools and services.
-SOC and NOC
Maintenance was completed as of 12:30AM.
Tonight at midnight, we will be performing a second attempt at the planned migration of hardware serving vpn.sonic.net in our Santa Rosa datacenter. Expected downtime for VPN access is expected to be 30 minutes or less.
-Matt
Update: As of 9:35pm, the repair work is complete. The damage was greater than initially anticipated, but all customers are now back online. Affected customers may have to reboot local equipment to restore connectivity. Thank you for your patience.
—
Early yesterday evening, there was a transformer fire in the Richmond District of San Francisco at 15th & Geary. Unfortunately, the fire impacted our infrastructure in the area, and some customers may have subsequently experienced limited, or no internet connectivity.
We have field engineers on site working to replace and repair the damaged portion of our network as quickly as possible. We believe the damage is limited in scope, and the current estimated time of repair is 5pm, at which point all affected customers should have their service restored.
If you have any questions in the meantime, or if your service is not restored by the estimated time of repair, please give support a call at your earliest convenience. Support can be reached at 1-888-766-4233, seven days a week, from 8am to 10pm.
We appreciate your patience in addressing this issue, and hope to have you up and running soon.
-Dan
Update (1:04AM): Maintenance complete.
Update: This Maintenance has been postponed to tomorrow night at midnight.
Beginning tonight at midnight, we will be performing upgrades on equipment serving Fusion Fiber customers in the Richmond District of San Francisco. Expected downtime for this operation is approximately 10 minutes.
-Network Engineering
Today, March 19th, starting at 2:00am, equipment serving a small subset of FlexLink Long Range customers in the Oakland area failed. We are dispatching to replace the affected equipment immediately. Expected time of repair 5:30am.
Update(4:45am): Affected hardware replaced and service has been restored
-Tomoc