Tuesday, January 12, starting at 11:59pm, we will be performing router maintenance on core equipment in San Jose. Maintenance is expected to take around 30 minutes and should not cause any customer impact.
-Tomoc
System Maintenence
UPDATE: All work is complete.
Starting at 11:59pm we will be running updates on custemer facing and backend servers. The updates are expected to be complete by 2am. Durning this time there may be brief interuptions to the following services:
- SMTP/POP3/IMAP
- FTP
- Membertools
- IPv6 Tunnels
We will post an update when the work is complete.
Grant and SOC
(Northern California) Legacy DSL Maintenance
Update (1:40AM): This maintenance is now complete.
Beginning tonight at midnight we will be performing maintenance on equipment serving legacy DSL customers. This maintenance can last for a couple hours but actual downtime for any affected customers shouldn’t last for more than 30 minutes at a time.
- Robbie
Non Intrusive Voice Maintenance
Tonight starting at midnight we will be conducting maintenance upgrades to core voice equipment. During this time people may experience transient issues with phone call completion, but the likelihood is rare. Â We will update when the work is compete. Â Estimated time of completion is 3am.
Thank you
Update: Maintenance is now complete, thanks
-Brandon
(Northern California) Legacy DSL Outage
Update (1:40AM): The failed hardware has been replaced and any affected customers should now be back online.
A portion of customers on our legacy DSL platform in northern California are currently offline due to what appears to be a hardware failure. We are investigating and will provide an update asap.
- Robbie
(Oakland) Intrusive Network Maintenance
Update(12:40AM): This maintenance is now complete.
We will be performing maintenance on a router in the Oakland area tonight beginning around midnight until 1am. During this maintenance window there will be a disruption of service for the customers that are homed off this router lasting around 15 minutes.
- Robbie
ASE Network Packet Loss
Currently Business Customers on ATT’s ASE Fiber network are experiencing packet loss.  We are on the phone with ATT working with them to resolve the issue.  We will provide an update once we have more information, thank you.
-Brandon
Update 10:42pm 12/16/2015
AT&T has isolated the issue and repaired accordingly. This issue should be resolved. Please contact Sonic NOC with any concerns of lingering packet loss if experienced.
-Michael Tardif
Support Closure for Holiday Party
Sonic will be closing early today, 12/11/2015 from 6pm to 10pm for our annual staff holiday party. We will return Saturday to our normal working hours. Our 24×7 network operations team will still be on call for our Enterprise, dedicated circuit and colocation customers.
Fusion/FlexLink Intrusive Maintenance – Santa Rosa
Update(2:55AM): This maintenance is now complete.
Update: This maintenance has been postponed until Monday, December 7th.
Beginning tonight at midnight we will be performing maintenance on equipment serving a portion of Fusion and FlexLink customers in the Santa Rosa area. Downtime for this operation can be up to a few hours in some cases, but is generally under 45 minutes for most affected customers.
– Robbie and Michael
Email Storage Cluster Event, Planned Maintenance
One of the two pairs of storage clusters we use for email storage had a cluster takeover event at 11:09 this AM. One of the heads lost communication with all of its disks and its partner successfully took over all of its services without any interruption. However, while tracing the failure of the first head to a failed FCAL optics package – which was replaced – the cluster interconnect adapter in the second head locked up and triggered a panic. This panic lead to a brief interruption if POP/IMAP services at approximately 12:00. The second filer rebooted successfully still in partner takeover.
Unfortunately, since the second filer is still in takeover mode and can’t see the cluster interconnect adapter, the most conservative resolution requires that we halt the second filer and replace the failed adapter. We’ve tentatively scheduled this for tomorrow after midnight provided that we receive the replacement adapter from our vendor in time. POP/IMAP services will be offline for the duration of the maintenance which should take less than an hour to complete.
-Kelsey and William
Update 01:00: The cluster interconnect adapter has been replaced and all services have been fully restored.
-Kelsey and William