San Francisco Fusion Fiber Outage

As of around 4:15 this afternoon, we encountered a widespread issue impacting San Francisco area Fusion Fiber circuits.  During this time, customers may have experienced reduced speeds and intermittent connectivity.  We identified hardware suspected to be responsible for the issues and performed emergency maintenance.  The vendor has been contacted for further investigation into the outage.  Connectivity appears be stabilized for all affected customers, and we are closely monitoring the situation.  Additional preventative maintenance may be performed tonight at 12:00 midnight, if deemed necessary.

-Sonic NOC

Sonic Telecom Core Maintenance

Update (12:35AM): Maintenance complete

Beginning tonight at midnight, we will be performing software upgrades on core network equipment in Mendocino, CA. Because traffic is expected to route around the affected equipment, no customer downtime is expected.

-Network Engineering

Santa Rosa Data Center UPS Maintenance

Update: Our vendor has had to reschedule yet again, this time for May 16th and 10AM.

Update: Maintenance has been rescheduled again, this time for May 1st.

Update: Maintenance has been rescheduled for April 27th.

Update: This maintenance has been cancelled and will be rescheduled at a later date.

Tomorrow, starting at 10:00, the UPSes that service our Santa Rosa data center will be undergoing scheduled preventative maintenance.  This includes overhauling one of them with a complete capacitor replacement.  The maintenance is fully scripted with our vendor and no interruption of power services to the datacenter are expected.  -Sonic Operations and Facilities

Sonic Telecom Core Maintenance

Update (12:18AM): Maintenance complete

Beginning tonight at midnight, we will be performing software upgrades on core network equipment in Santa Rosa, CA. Because traffic is expected to route around the affected equipment, no customer downtime is expected.

-Network Engineering

Sonic Telecom Core Maintenance

Update (12:57AM): Maintenance complete

Beginning tonight at midnight, we will be performing software updates on core network equipment. Traffic is expected to route around the affected equipment, but customers may experience a brief period of routing instability as traffic shifts.

-Network Engineering

Sonic SSE Fiber Disturbance

Today at approximately 2:05 PM a set of our Sonic SSE Fiber customers began experiencing traffic throughput issues and packet loss. We identified the cause of this disturbance as a byproduct of an unexpected network build provided by our fiber vendor. We promptly mitigated the issue and we have taken preventative measures to avoid this disturbance in the future.

Virtualization infrastructure upgrade

UPDATE: upgrade complete.

 

Beginning at 11:59pm our Operations team will be upgrading the switches in our virtualization cluster. This may lead brief interruptions of tools and services.

 

-SOC and NOC

VPN Concentrator Maintenance

Maintenance was completed as of 12:30AM.

 

Tonight at midnight, we will be performing a second attempt at the planned migration of hardware serving vpn.sonic.net in our Santa Rosa datacenter.  Expected downtime for VPN access is expected to be 30 minutes or less.

-Matt

San Francisco (Richmond District) Fusion Intrusive Maintenance

Update: As of 9:35pm, the repair work is complete. The damage was greater than initially anticipated, but all customers are now back online. Affected customers may have to reboot local equipment to restore connectivity. Thank you for your patience.

—

Early yesterday evening, there was a transformer fire in the Richmond District of San Francisco at 15th & Geary. Unfortunately, the fire impacted our infrastructure in the area, and some customers may have subsequently experienced limited, or no internet connectivity.

We have field engineers on site working to replace and repair the damaged portion of our network as quickly as possible. We believe the damage is limited in scope, and the current estimated time of repair is 5pm, at which point all affected customers should have their service restored.

If you have any questions in the meantime, or if your service is not restored by the estimated time of repair, please give support a call at your earliest convenience. Support can be reached at 1-888-766-4233, seven days a week, from 8am to 10pm.

We appreciate your patience in addressing this issue, and hope to have you up and running soon.

-Dan