Category: Uncategorized

On September 28th and 29th one of the UPSes that serves our Santa Rosa datacenter and colo facility will have all of it’s batteries replaced.  (Over 4,000 pounds of them!)  We will also be taking the opportunity to install a battery monitoring system in this, our oldest, UPS.  Once installed, the battery monitoring system will allow us to individually monitor each battery in the UPS allow us to predict pending failure as well as reliably detect other faults.  Service will not be interrupted, this is a fully scripted event coordinated with our vendors.

This evening, at 12:01AM, we will be performing intrusive maintenance work on equipment serving Fusion and FlexLink Ethernet customers in Healdsburg. Affected customers will experience up to 10 minutes of downtime while this work is performed.

-Tim

We lost connectivity to one of our Customer MySQL servers at approximately 0130 this morning, service was restored at 0213, and we are investigating the cause of the loss of service. We apologize for any inconvenience this may have caused.

This evening at 9:38PM one of our DSL aggregation routers that serves DSL to the Bay Area locked up and stopped forwarding traffic. We reloaded the router remotely and it is now functioning normally. We will be monitoring this router for any further issues and will be replacing it if further problems are expected.

-Jared

At 4:10PM, we suffered a line card failure on equipment serving a number of Fusion customers served out of the SNFCCA12 CO in San Francisco. We are currently dispatching a technician and will provide updates as we work to resolve this outage.

-Tim, Nathan, and Jared

Update: We have determined that the line card did not actually fail and all affected customers should be back online. We will continue to monitor for any further issues.

Update – 8/24 12:45AM: After running normally for hours, the line card again began exhibiting strange behavior. It has been replaced with a new unit. We apologize for any inconvenience caused by this outage.

This evening starting at 10:00PM, we will be performing maintenance on CLEC equipment serving eastern Santa Rosa. Expected downtime is less than 15 minutes.

This evening, at 12:01AM, we will be performing minor maintenance on one of our ATM switches. The work should only take up to 10 minutes and we do not expect any impact on customer traffic.

-Tim and Jared

This morning our phone lines in to the Sonic.net office are reporting all circuits busy. We are actively working with our vendor who has not yet diagnosed the trouble.

This outage affects inbound and outbound calls through our office phone system. No other services are affected by this outage.

As always, we are are available via email to support@sonic.net.

–Update:  This problem was due to a loss of signaling on our providers network.  The issue has been repaired and our support department is open and taking calls at this time.  Thank you for your patience.

We have been receiving reports this morning of issues reaching hosts in the Sonic network from external sources. We are currently investigating this issue and will provide more information once it becomes available.

-Tim, Nathan, and the NOC

Update: We have narrowed down the problem to our upstream provider Level 3. There was a routing issue inside of their network that has since been resolved. We have confirmed that any connectivity issues should be resolved at this point.

This evening, at 12:01AM, we will be performing backbone and intrusive power maintenance on equipment at the San Rafael (SNRFCA01) central office. We expect the backbone maintenance to have no impact on customer traffic. The power maintenance will cause a brief interruption of service for FlexLink Long Range customers.

Beginning at 2:00AM, we will be performing power maintenance on equipment at the Burlingame (BRLNCA01) central office. We expect this work to cause up to 30 minutes of downtime for Fusion and FlexLink Long Range customers.

-Tim, Jared, Clay, and Monroe