UPDATE: Complete
Tonight at 11:59pm we will be running software updates on multiple customer facing systems. The following services may experience brief outages:
- SMTP/POP3/IMAP
- Customer Web Hosting
- Membertools
- Webmail
- IPv6 Tunnels
- VPN
-SOC
SMTP maintenance
UPDATE: complete.
Our SMTP servers will be under maintenance tonight to receive software updates and reissued SSL certificates. There may be brief interruptions to outgoing mail starting now and lasting a few minutes.
-SOC
San Francisco Fusion Fiber Outage
As of around 4:15 this afternoon, we encountered a widespread issue impacting San Francisco area Fusion Fiber circuits. During this time, customers may have experienced reduced speeds and intermittent connectivity. We identified hardware suspected to be responsible for the issues and performed emergency maintenance. The vendor has been contacted for further investigation into the outage. Connectivity appears be stabilized for all affected customers, and we are closely monitoring the situation. Additional preventative maintenance may be performed tonight at 12:00 midnight, if deemed necessary.
-Sonic NOC
Sonic Telecom Core Maintenance
Update (12:35AM): Maintenance complete
Beginning tonight at midnight, we will be performing software upgrades on core network equipment in Mendocino, CA. Because traffic is expected to route around the affected equipment, no customer downtime is expected.
-Network Engineering
Santa Rosa Data Center UPS Maintenance
Update: Our vendor has had to reschedule yet again, this time for May 16th and 10AM.
Update: Maintenance has been rescheduled again, this time for May 1st.
Update: Maintenance has been rescheduled for April 27th.
Update: This maintenance has been cancelled and will be rescheduled at a later date.
Tomorrow, starting at 10:00, the UPSes that service our Santa Rosa data center will be undergoing scheduled preventative maintenance. This includes overhauling one of them with a complete capacitor replacement. The maintenance is fully scripted with our vendor and no interruption of power services to the datacenter are expected. -Sonic Operations and Facilities
Sonic Telecom Core Maintenance
Update (12:18AM): Maintenance complete
Beginning tonight at midnight, we will be performing software upgrades on core network equipment in Santa Rosa, CA. Because traffic is expected to route around the affected equipment, no customer downtime is expected.
-Network Engineering
Sonic Telecom Core Maintenance
Update (12:57AM): Maintenance complete
Beginning tonight at midnight, we will be performing software updates on core network equipment. Traffic is expected to route around the affected equipment, but customers may experience a brief period of routing instability as traffic shifts.
-Network Engineering
Sonic SSE Fiber Disturbance
Today at approximately 2:05 PM a set of our Sonic SSE Fiber customers began experiencing traffic throughput issues and packet loss. We identified the cause of this disturbance as a byproduct of an unexpected network build provided by our fiber vendor. We promptly mitigated the issue and we have taken preventative measures to avoid this disturbance in the future.
Virtualization infrastructure upgrade
UPDATE: upgrade complete.
Beginning at 11:59pm our Operations team will be upgrading the switches in our virtualization cluster. This may lead brief interruptions of tools and services.
-SOC and NOC
VPN Concentrator Maintenance
Maintenance was completed as of 12:30AM.
Tonight at midnight, we will be performing a second attempt at the planned migration of hardware serving vpn.sonic.net in our Santa Rosa datacenter. Expected downtime for VPN access is expected to be 30 minutes or less.
-Matt