Currently PG&E utility is offline at the 200 Paul San Francisco facility, and it is running normally on generator. Automatic transfer switching worked as designed, and all is well.
After the failure last week of a transfer switch in this San Francisco datacenter, it’s good to see that repairs to that redundant power system worked and that the repaired transfer switch did its job.
-Dane & Nathan
At 7:11PM tonight one of our two Core4 CRAC (Computer Room Air Conditioning) units unexpectedly shut itself down. Nothing instills fear more than receiving pages titled “Sys A Enable Switch Turned Off / Service Now” and “High Discharge Air Temperature” in rapid succession. After the initial panic passed, it was clear that all redundant systems we operating correctly and the second system had responded correctly and ramped up to handle the total cooling load. Once on site, there was no outward indication why System A had shutdown as the system enable switch was correctly in the “On” position and it had supply power. However, upon further investigation, it was apparent that the enable switch was water-logged, oxidized and shorted-out, signaling the system to shutdown. The switch has been removed from service and both systems are 100% operational again.
Although it is disappointing to see a system failure caused by something as simple as an improperly weatherproofed mechanical room and control panel, it is rewarding to see our commitment to and investment in redundancy pay off. And, ultimately, that the prototype Core4 CRAC system behaved as expected.
Special thanks goes out to Jimmy and Kent of Bell Products who interrupted their dinners to come out and verify that all systems were functioning correctly.
-Kelsey, Nathan and Russ
One of our backbone network transport links began having issues this morning. We have removed that link from service, and are routing traffic internally around the problem as we work with the transport circuit provider to diagnose the intermittent problems. These problems did not cause any customer impact, but as we route around the problem, customers may notice sub-optimal paths inside Sonic’s network (i.e. from San Francisco to San Jose via Santa Rosa).
We are keeping a close eye on the situation and will restore normal routing once we are certain that the transport circuit is fully resolved.
-Jared and Nathan
Earlier today we migrated the remaining humboldt1.com Internet services over to Sonic.net.
This will provide those customers with improved service and reliability that all Sonic.net customers enjoy.
If you are a former Humboldt Internet customer and you are having trouble checking your e-mail or your website, please contact our Support team :
http://www.sonic.net/contactus.shtml
1 (707) 547-3400
support@sonic.net
–Augie
At this time we are tracking an ATM backhaul problem affecting LATA9. This LATA covers DSL service for the Stockton/Modesto areas. We believe the problem to be on AT&T’s backhaul side, and are working with them currently to diagnose and repair.
-Jared
Update: We have narrowed the issue down to a specific line card in an ATM device in our network. Due to the nature of the problem, we had to perform a reload of the entire device. This briefly affected customers in LATAs 1, 2, and 6 as well. We apologize for affecting previously unaffected customers while reloading this device.
This evening, at 12:01AM, we will be performing a software upgrade on the DSLAM that serves Fusion and FlexLink customers in the downtown Santa Rosa area. Expected downtime is less than 30 minutes while the DSLAM is rebooted onto the new software release.
-Tim and Nathan
This evening, Saturday, February 6 at 12:01 AM we will be performing maintenance reloads of two Redbacks that terminate traditional DSL service. This will affect all Chico DSL subscribers, and some of our Bay Area DSL subscribers. Expected downtime is 5 minutes.
-Jared
Update: The reloads have been completed without incident. All affected customers should be back online at this time. Total downtime was less than 5 minutes.
This evening, between 12:01AM and 1:00AM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink ADSL2+ customers in downtown Santa Rosa. Affected customers should experience less than 15 minutes of downtime.
-Tim
Both SquirrelMail and NutsMail we’re unavailable from approximately 11:45 to 12:00 today. AtMail was not affected. We’re sorry for the interruption and will review the procedures that led to the failure.
-Sonic Operations
We are currently experiencing issues with one of the AT&T back haul circuits that carries traffic to and from the Sonic Telecom network. We are working with AT&T on resolving the issue and have isolated the trouble to a specific card in a central office. We will provide further updates as soon as possible.
-Tim and Nathan
Update: The issue has been resolved. The circuit has remained stable for the past 45 minutes. We will continue to monitor closely for any further problems.