DNS Policy Changes Deployed

We’ve completed the deployment of our new DNS policies as covered in the Upcoming DNS changes MOTD.  To reiterate, we’ve disabled access to our recursive servers from off our network, are enforcing DNSSEC validation and have enabled two commercial RPZ lists to help protect our customers from phishing, viruses and malware.  For more information please see this forum post.

Update:  We disabled one of the commercial RPZ lists earlier this afternoon.  Despite substantial testing and review prior to deploying this to our customers that the service – at least as it stands now – is overly aggressive in its listing policy.

-Kelsey and Augie

DNS resolution problem.

Today at 3:45pm PST we suffered a DNS outage to one of our two DNS clusters, service was restored to the cluster at 3:51pm.

All services are functioning as expected at this time, and we are investigating the cause of the outage.

–Augie

Legacy DSL DHCP Maintenance

Tonight, at midnight, we will reboot one of our DHCP servers that handles legacy DSL customers terminated in San Francisco onto a new kernel.  The server should be down for only a few minutes and little to no interruption of services should occur for the 2,700 affected customers.  This update will resolve a problem with an Ethernet transceiver lockup affecting one of the servers redundant uplinks.

-Kelsey

Update Wed Mar  6 00:14:32 PST 2013,  maintenance complete.  Total downtime for the server was less that 5 minutes.

Telecom Network Maintenance

Tonight, March 4, starting at 11:30pm, we will be performing maintenance on equipment serving our telecom network.  We do not anticipate any customer impact during this maintenance window.

Update: This maintenance window has been postponed until 11:30pm tomorrow night, March 5.

Update: All maintenance completed as planned, no customer impact.

-Tomoc and Nathan

MySQL Maintanance

UPDATE: All scheduled maintenance has been completed, and service is now fully restored.

 

UPDATE: We will also be performing maintenance on several customer facing services starting at 11:59 pm until 1am. Services include:

Imap/pop3

Customer web servers

Internal MySQL servers

Downtime should be minimal for these services.

Tonight, February 28th, 2013, at 11:59PM, we will be performing a replacement and upgrade to our customer-facing MySQL server. The work should take approximately 30 minutes. During the maintenance window, customer databases will not be available.

— Joe, Grant and the SOC

ATM Outage

This morning, February 27, at 12:11AM, an ATM aggregation circuit serving FRATM and legacy DSL in the Chico area went down. We are currently working with our vendor to bring this circuit back up as soon as possible. Update to follow.

Update: Circuit restored at 1:45am. We are working with our vendor to ensure this does not happen again.

-Tomoc

Upcoming DNS changes

On Tuesday February 26th we will begin the roll out of several security enhancements to our recursive DNS servers.  These features will keep both our customers and our systems more secure and limit abuse of our resources. The changes include:

Enabling DNSSEC validation.
Enabling 2 commercial DNS RPZ services.
Closing our DNS servers to off network requests.

These changes will be rolled out to customers served out of Los Angeles first.  Provided that we don’t run into any major issues, the remaining systems serving the rest of our customers will be migrated on March 5th.

Due to the potentially controversial nature of using DNS RPZ, a set of alternate servers are available to customers who wish to opt out.  For more information please visit the following Forums posting:

https://forums.sonic.net/viewtopic.php?f=5&t=1186

Update 26, February — Stage one is complete, Los Angeles customers are now using the new DNS services.

Fusion/FlexLink Maintenance

Tonight, February 10, starting at 11:59pm, we will be performing maintenance on equipment serving a subset of Fusion/FlexLink customers in the Berkeley area. Customers may experience down time of up to 30 minutes.

Update: Apologies for the incorrect date, this will be commencing 11:59pm, Februaury 13

Update: Maintenance completed as planned, customer downtime was less than 5 minutes

-Tomoc and Clay

Network Maintenance

Tonight, February 12, starting at 11:59pm, we will be performing maintenance on equipment in the North Bay and Los Angeles areas. Customers may experience brief network instability to certain destinations.

Update: Maintenance is now complete. If you experience any problems with your connection, please contact support as usual.

-Tomoc Nathan and Robbie