At 4:15pm, a small subset of legacy DSL subscriber connections became unstable. We are investigating the cause of this issues and are working to stabilize the affected circuits as fast as possible.
Update: All affected subscribers have been migrated to new equipment. Affected customers may have noticed several service interruptions lasting no more than 3 minutes each. Please note that under some circumstances, rebooting the DSL equipment on site may be required.
-Tomoc and Robbie
Over the past few days we’ve seen a massive increase in both the number and volume of DNS Amplification Attacks using our recursive name servers. This is likely due to the fact that our new name servers provide more verbose answers and are therefore amplify traffic more effectively than our old servers. We unfortunately had to roll back blocking off-net use of our recursive servers and blocking these requests entirely is not currently an option at this time. To mitigate the effects of the attacks both on our systems and their targets, we’ve instituted rate limits on the total number of queries per second any given IP address is able to source to our servers. The rate limits are high enough that they should not interfere with any normal (and acceptable) use. However, it is possible that a customer doing bulk DNS lookups (such as log processing or running a busy mail server) may run into issues and experience intermittent delays resolving host names.
-Kelsey, Augie and Nathan
We are performing an emergency reload on equipment serving a subset of legacy DSL, Business-T, and FRATM customers in the Bay Area and Modesto/Stockton area. We will send another update shortly once the maintenance has completed.
-Tim and the NOC
Update: Service for affected customers should now be restored. At around 3:03PM, a controller card in one of our aggregation switches performed an automatic failover to the standby card. For unknown reasons, the standby card was not operating correctly once it took over, causing an outage for some percentage of subscribers. We were forced to perform a full reset of the device to restore service. We are investigating both the cause of the failover as well as the additional issues experienced afterwards and we apologize for the duration of this outage.
This evening, beginning at 11:00PM, we will be performing invasive maintenance on equipment serving a subset of ATM subscribers in the Bay Area and Sacramento. This will affect Business-T, FRATM, and legacy DSL services specifically. We expect downtime for the affected customers to be less than 15 minutes.
-Tim, Robbie, and Tomoc
Update: The maintenance work has been completed as planned. All affected customers should be back online.
Tonight, March 7, starting at 11:30pm, we will be performing maintenance on equipment serving a small subset of legacy DSL customers in the Bay Area. Expected customer down time is less than 30 minutes.
Update: Maintenance complete. Affected customers may need to restart their DSL equipment.
-Tomoc
We’ve completed the deployment of our new DNS policies as covered in the Upcoming DNS changes MOTD. To reiterate, we’ve disabled access to our recursive servers from off our network, are enforcing DNSSEC validation and have enabled two commercial RPZ lists to help protect our customers from phishing, viruses and malware. For more information please see this forum post.
Update: We disabled one of the commercial RPZ lists earlier this afternoon. Despite substantial testing and review prior to deploying this to our customers that the service – at least as it stands now – is overly aggressive in its listing policy.
-Kelsey and Augie
Tonight, March 4, starting at 11:30pm, we will be performing maintenance on equipment serving our telecom network. We do not anticipate any customer impact during this maintenance window.
Update: This maintenance window has been postponed until 11:30pm tomorrow night, March 5.
Update: All maintenance completed as planned, no customer impact.
-Tomoc and Nathan
UPDATE: All scheduled maintenance has been completed, and service is now fully restored.
UPDATE: We will also be performing maintenance on several customer facing services starting at 11:59 pm until 1am. Services include:
Imap/pop3
Customer web servers
Internal MySQL servers
Downtime should be minimal for these services.
Tonight, February 28th, 2013, at 11:59PM, we will be performing a replacement and upgrade to our customer-facing MySQL server. The work should take approximately 30 minutes. During the maintenance window, customer databases will not be available.
— Joe, Grant and the SOC
This morning, February 27, at 12:11AM, an ATM aggregation circuit serving FRATM and legacy DSL in the Chico area went down. We are currently working with our vendor to bring this circuit back up as soon as possible. Update to follow.
Update: Circuit restored at 1:45am. We are working with our vendor to ensure this does not happen again.
-Tomoc
Tonight, February 10, starting at 11:59pm, we will be performing maintenance on equipment serving a subset of Fusion/FlexLink customers in the Berkeley area. Customers may experience down time of up to 30 minutes.
Update: Apologies for the incorrect date, this will be commencing 11:59pm, Februaury 13
Update: Maintenance completed as planned, customer downtime was less than 5 minutes
-Tomoc and Clay