This Wednesday (24, Feb.) morning at 12:01 AM the Secure Server Services will be un-available for approximately 30 minutes; this affects all customers with services on ssl.sonic.net or secure web hosting services with us.
After the maintenance these customers will see a substantial performance boost.
–Augie
We’re currently experiencing a rather large outage in San Francisco, presumably caused by a power failure to some of our colocation space there. All available resources are being brought to bear, and we’ll have more shortly. This outage is likely impacting a large portion of our DSL customers as well as some Dial, Biz-T, FRATM, and other services. -Nathan, Jared, and the rest of the NOC
Update 8:25AM: We have confirmed that the issue is caused by a power failure. Apparently the facility at 200 Paul Avenue is currently without utility power. The vast majority of the site’s generators, ATSes, and UPS systems worked properly, but the UPSes feeding suite 502 (the location of some of our equipment at that facility) did not transfer correctly. More to follow.
Update 9:19AM: Power has been restored to our equipment, and we’re ensuring that everything returns to service cleanly.
Update 9:32AM: Service to the vast majority of our customers has been restored. There appears to be one DSL aggregation device that is still having trouble — we’re taking a look at that now.
Update 10:22AM: All services have been restored. Please let us know if you still have any outstanding issues. We’ll be having a serious chat with our colocation provider about this event, as this outage occurred despite a massive investment in fully redundant A+B power — both of which died simultaneously.
Earlier today we migrated the remaining humboldt1.com Internet services over to Sonic.net.
This will provide those customers with improved service and reliability that all Sonic.net customers enjoy.
If you are a former Humboldt Internet customer and you are having trouble checking your e-mail or your website, please contact our Support team :
http://www.sonic.net/contactus.shtml
1 (707) 547-3400
support@sonic.net
–Augie
At approximately 3:00 AM today, one of our DSL aggregation routers serving customers in the Bay Area suffered what appears to be a software crash. This software crash kept the router out of service until 3:14 AM, at which time it was booted up and handling customer traffic normally.
The exact cause of the software crash is being investigated with the vendor at this time. We are monitoring the router, and all affected customers appear to be back online at this time.
-Jared
Update: The router has unexpectedly crashed again. We are in the process of moving all customers served by this router to a hot spare. Connectivity should be restored shortly.
Update: All affected customers have been moved to the hot spare successfully. All affected customers should be online again at this time.
At this time we are tracking an ATM backhaul problem affecting LATA9. This LATA covers DSL service for the Stockton/Modesto areas. We believe the problem to be on AT&T’s backhaul side, and are working with them currently to diagnose and repair.
-Jared
Update: We have narrowed the issue down to a specific line card in an ATM device in our network. Due to the nature of the problem, we had to perform a reload of the entire device. This briefly affected customers in LATAs 1, 2, and 6 as well. We apologize for affecting previously unaffected customers while reloading this device.
This evening, at 12:01AM, we will be performing a software upgrade on the DSLAM that serves Fusion and FlexLink customers in the downtown Santa Rosa area. Expected downtime is less than 30 minutes while the DSLAM is rebooted onto the new software release.
-Tim and Nathan
This evening, Saturday, February 6 at 12:01 AM we will be performing maintenance reloads of two Redbacks that terminate traditional DSL service. This will affect all Chico DSL subscribers, and some of our Bay Area DSL subscribers. Expected downtime is 5 minutes.
-Jared
Update: The reloads have been completed without incident. All affected customers should be back online at this time. Total downtime was less than 5 minutes.
This evening, between 12:01AM and 1:00AM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink ADSL2+ customers in downtown Santa Rosa. Affected customers should experience less than 15 minutes of downtime.
-Tim
Both SquirrelMail and NutsMail we’re unavailable from approximately 11:45 to 12:00 today. AtMail was not affected. We’re sorry for the interruption and will review the procedures that led to the failure.
-Sonic Operations
We are currently experiencing issues with one of the AT&T back haul circuits that carries traffic to and from the Sonic Telecom network. We are working with AT&T on resolving the issue and have isolated the trouble to a specific card in a central office. We will provide further updates as soon as possible.
-Tim and Nathan
Update: The issue has been resolved. The circuit has remained stable for the past 45 minutes. We will continue to monitor closely for any further problems.