On September 28th and 29th one of the UPSes that serves our Santa Rosa datacenter and colo facility will have all of it’s batteries replaced. (Over 4,000 pounds of them!) We will also be taking the opportunity to install a battery monitoring system in this, our oldest, UPS. Once installed, the battery monitoring system will allow us to individually monitor each battery in the UPS allow us to predict pending failure as well as reliably detect other faults. Service will not be interrupted, this is a fully scripted event coordinated with our vendors.
CLEC Intrusive Maintenance
This evening, at 12:01AM, we will be performing intrusive maintenance work on equipment serving Fusion and FlexLink Ethernet customers in Healdsburg. Affected customers will experience up to 10 minutes of downtime while this work is performed.
-Tim
Customer MySQL server down-time.
We lost connectivity to one of our Customer MySQL servers at approximately 0130 this morning, service was restored at 0213, and we are investigating the cause of the loss of service. We apologize for any inconvenience this may have caused.
VPN Concentrator Certificate Update
Today the security certificate used by our VPN concentrator expired, and we have replaced that certificate with a new one. The certificate is used by the Cisco VPN Client to properly secure VPN connections, so clients with the old certificate will no longer be able to connect. You can get the new certificate by installing the latest version of our VPN Client at this URL:
http://sonic.net/support/hotspotportal/
-Jared and Kelsey
UPS Maintenance
Next Tuesday, 9/14, the UPS’s serving our Santa Rosa data center will be be undergoing their semi-annual non-intrusive maintenance and inspection. This is a fully scripted event coordinated with our service vendors and no interruption of power services should occur. -Kelsey and Russ
DSL Aggregation Issues
This morning our support group has been tracking an issue that is affecting DSL subscribers connected to one of our DSL aggregation routers. We are investigating possible hardware or physical problems that may be affecting that router.
-Jared
Update: The problem was tracked down to a failing port on one of our ATM switches, which was causing heavy packet loss. We moved our connection off that port and the problem should now be resolved.
CLEC Intrusive Maintenance – SNFCCA21
At 12:01AM on 9/2/10 we will be performing intrusive maintenance on the DSLAM serving the (San Francisco) SNFCCA21 POP. All Flexlink Ethernet and Fusion customers served from this POP should experience between 5-15 minutes of downtime while the work is performed.
-Matt, Tim and Jared
Update: Maintenance work has been completed. Downtime for affected customers was less than 5 minutes.
DSL Aggregation Router Lockup
This evening at 9:38PM one of our DSL aggregation routers that serves DSL to the Bay Area locked up and stopped forwarding traffic. We reloaded the router remotely and it is now functioning normally. We will be monitoring this router for any further issues and will be replacing it if further problems are expected.
-Jared
Fusion Outage
At 4:10PM, we suffered a line card failure on equipment serving a number of Fusion customers served out of the SNFCCA12 CO in San Francisco. We are currently dispatching a technician and will provide updates as we work to resolve this outage.
-Tim, Nathan, and Jared
Update: We have determined that the line card did not actually fail and all affected customers should be back online. We will continue to monitor for any further issues.
Update – 8/24 12:45AM: After running normally for hours, the line card again began exhibiting strange behavior. It has been replaced with a new unit. We apologize for any inconvenience caused by this outage.
CLEC Maintenance
This evening starting at 10:00PM, we will be performing maintenance on CLEC equipment serving eastern Santa Rosa. Expected downtime is less than 15 minutes.