Month: February 2013

MySQL Maintanance

UPDATE: All scheduled maintenance has been completed, and service is now fully restored.

 

UPDATE: We will also be performing maintenance on several customer facing services starting at 11:59 pm until 1am. Services include:

Imap/pop3

Customer web servers

Internal MySQL servers

Downtime should be minimal for these services.

Tonight, February 28th, 2013, at 11:59PM, we will be performing a replacement and upgrade to our customer-facing MySQL server. The work should take approximately 30 minutes. During the maintenance window, customer databases will not be available.

— Joe, Grant and the SOC

ATM Outage

This morning, February 27, at 12:11AM, an ATM aggregation circuit serving FRATM and legacy DSL in the Chico area went down. We are currently working with our vendor to bring this circuit back up as soon as possible. Update to follow.

Update: Circuit restored at 1:45am. We are working with our vendor to ensure this does not happen again.

-Tomoc

Upcoming DNS changes

On Tuesday February 26th we will begin the roll out of several security enhancements to our recursive DNS servers.  These features will keep both our customers and our systems more secure and limit abuse of our resources. The changes include:

Enabling DNSSEC validation.
Enabling 2 commercial DNS RPZ services.
Closing our DNS servers to off network requests.

These changes will be rolled out to customers served out of Los Angeles first.  Provided that we don’t run into any major issues, the remaining systems serving the rest of our customers will be migrated on March 5th.

Due to the potentially controversial nature of using DNS RPZ, a set of alternate servers are available to customers who wish to opt out.  For more information please visit the following Forums posting:

https://forums.sonic.net/viewtopic.php?f=5&t=1186

Update 26, February — Stage one is complete, Los Angeles customers are now using the new DNS services.

Fusion/FlexLink Maintenance

Tonight, February 10, starting at 11:59pm, we will be performing maintenance on equipment serving a subset of Fusion/FlexLink customers in the Berkeley area. Customers may experience down time of up to 30 minutes.

Update: Apologies for the incorrect date, this will be commencing 11:59pm, Februaury 13

Update: Maintenance completed as planned, customer downtime was less than 5 minutes

-Tomoc and Clay

Network Maintenance

Tonight, February 12, starting at 11:59pm, we will be performing maintenance on equipment in the North Bay and Los Angeles areas. Customers may experience brief network instability to certain destinations.

Update: Maintenance is now complete. If you experience any problems with your connection, please contact support as usual.

-Tomoc Nathan and Robbie

Fusion/FlexLink Equipment Maintenance

Tomorrow, February 8, starting at 5:00am, we will be performing maintenance on equipment serving a subset of Fusion/FlexLink customers in the Berkeley area. Customers may experience reachability issues for several seconds during this operation.

Update: Due to unforeseen circumstances, this maintenance has been postponed

-Tomoc and Clay

Network Maintenance

Tonight, February 6, starting at 10pm, we will be performing maintenance on network infrastructure at PAIX in Palo Alto. Customers may experience brief routing instability as we shift traffic around our network.

Update: Maintenance was completed within expected parameters

-Nathan, Robbie and Tomoc

Problems with our Internal MySQL server

UPDATE –  At this time all services should be restored that were affected by this outage. We are investigating the root cause of this issue and will work to prevent similar problems in the future.

We are currently experiencing problems with out internal MySQL server. Member tools is not available at this time. We are currently working to restore service and should have the problem resolved shortly.

-William and the SOC

MySQL upgrade tonight

Update: Everything went well and all services should be available.

 

Tonight 2-1-2012 we will be upgrading our database servers. At 11:00 pm we will upgrade our internal MySQL server. This will cause member tools to be unavailable for approximately 5 minutes. Then at 11:59 pm we will upgrade both of our customer MySQL servers. They will be unavailable for use for approximately 5 minutes.

-William and the SOC