We are currently experiencing a power issue affecting some portion of our customers served out of our San Francisco point of presence. We are working with the facility to determine the cause of the issue and an estimated time of resolution. This outage is affecting Fiber DIA customers and may have caused some temporary DNS resolution issues for other customers. We will provide further updates shortly.

-Tim, Nathan and the NOC

Update: Power has stabilized as of around 10AM. All affected customers should be back up. We are working further with facility technicians to determine what caused the outage.

We are hearing scattered reports of reachability problems affecting a small
number of remote endpoints behind a specific service provider. Only a subset
of our network appears to be affected by this. We have been in touch with
this service provider, and they have diagnosed the issue to a bad line card
in a particular router. They are presently working with their vendor to get
that card repaired.

Due to the makeup of the Internet we’re unable to effectively re-route
traffic around this failing infrastructure. We’ll make sure that this
provider drives to resolution.

-Nathan and the NOC

Akamaihd.net and rackcdn.com both ended up one of the RPZ zones that we subscribe to this morning and were whitelisted by our staff as soon as it came to our attention.   These listings could have prevented the proper loading of many popular websites.  We’re working with our vendor to ensure that listing like this do not happen again.

-Sonic Ops Team

Sonic.net Support and Sales will close at 10:00pm Friday May 17th for our employee movie night. Support and Sales will open at the regular time Saturday morning.

– John F. and the Sonic.net Staff.

Tonight, starting at midnight, we’re going to apply kernel updates and reboot several systems, including several internal application and SQL servers, as well as public facing clusters that handle services like mail and webmail.  Overall impact should be minimal, but customers may experience delays or brief outages while accessing affected services while the systems are rebooted.

-Kelsey

While we’ve had ongoing issues with our recursive name servers being used as part of DNS amplification attacks for the past few months,  they were severe enough yesterday that even with the rate limits and other mitigation techniques we have had in place that normal usage and performance was affected.  We finally had to resort to blocking the most popular DNS queries used in the attacks in order to prevent any impact to our regular services.  Customers may have noticed slow DNS requests, most likely experienced as slow loading of web pages, off and on until early afternoon.  We expect that we will also finally block all off-net access to our recursive DNS servers sometime in the next few days.  Once complete, it should prevent this from being an issue moving forward.

In addition, we’re working on identifying our customers that appear to have zombied systems that are being used to participate in the bot-nets that are responsible for the attacks.

Sorry for the MOTD delay.

-Kelsey and William

This evening, beginning at 11:59PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink Ethernet customers in the Forestville area. Expected downtime for affected customers is less than 15 minutes.

-Tim

Update: Maintenance has been completed successfully.

This evening, beginning at 11:59PM, we will be performing maintenance on equipment serving a small subset of Legacy DSL subscribers in the Bay Area. Expected customer downtime is less than 15 minutes.

-Tim

Update: Maintenance has been completed as planned.

Tonight, April 24, starting at 11:59PM, we will be performing maintenance on equipment serving Sonic Telecom subscribers in northern California. We do not anticipate any customer down time for this maintenance window.

– Tomoc and Tim