Tomorrow evening (24, June) starting at 10pm Pacific Time we will be changing the power configuration on the NetApp storage filers that store customer e-mail; these filers are redundantly powered and we expect no down time for them during the maintenance window. During this time we will also perform server maintenance that may cause brief service interruption to our customer facing Member Tools.
–Augie
Since the cluster failure last month we had been running our rercursive name servers in a sub optimal (but still redundant) configuration. Last Friday, we completed the re-deployment and once again have a multiple layers of redundancy with both our primary and secondary recursive name servers announced into our network in Santa Rosa, San Francisco and Los Angeles. The old Alteon hardware load balancers that used to handle 208.201.224.11 in Santa Rosa and San Francisco have been replaced with Linux LVS/Quagga BGP servers that do not suffer from the same limitations that caused the original cluster failure. While each of the real servers could handle many times our total DNS load on its own, neither of the Alteons that were responsible for directing traffic for 208.201.224.11 could handle it after some routine network configuration shifted traffic inside of our network. Apart from a small hiccup due to a BGP misconfiguration Friday morning, the new clusters have performed perfectly. –Kelsey, Nathan and Jared
This Tuesday, June 22, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers in Sausalito, San Mateo, San Jose, and Millbrae. Affected customers should expect less than 30 minutes of downtime.
-Tim, Nathan, and Jacob
This evening, at 12:01AM, we will be performing a software upgrade on the DSLAM that serves Fusion and FlexLink customers in Los Altos. Expected downtime is less than 30 minutes while the DSLAM is rebooted onto the new software release.
-Nathan and Tim
Our authoritative name servers have had IPv6 addresses for some time now but until earlier today our glue records did not list the server’s AAAA addresses. Support for IPv6 on our authoritative name servers is now complete. However, it is worth noting that v6 requests account for <0.1% of our total authoritative DNS requests. Customer’s with their own registered name servers (you run your own DNS servers) that are IPv6 ready may submit requests to have their glue records manually updated to include their AAAA records. -Kelsey
This evening, at 12:01AM, we will be performing a software upgrade on the DSLAM that serves Fusion customers in Daly City. Expected downtime is less than 30 minutes while the DSLAM is rebooted onto the new software release.
-Nathan and Tim
This evening, at 12:01AM, we will be performing a software upgrade on the DSLAM that serves Fusion and FlexLink customers in Forestville. Expected downtime is less than 30 minutes while the DSLAM is rebooted onto the new software release.
-Nathan and Tim
Update: Everything went as planned. Customers experienced approximately 15 minutes of downtime.
This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving FlexLink Ethernet customers in Santa Rosa, Petaluma, and Rohnert Park. Affected customers should experience less than 10 minutes of downtime.
-Tim and Matt
This evening at 12:01Am we will be performing a card swap on one of the aggregation routers serving legacy T1s out of our Santa Rosa office. This should not be customer impacting, but like most maintenance there is a small potential chance of downtime.
-Matt Tim and Jared
Update: This router maintenance has completed without incident.
Tonight between 10 and 11PM we will be performing a card swap on a backbone router in our San Jose POP. Customer traffic should not be affected.
-Jared
Update: The card swaps have been done, and there was no interruption of service.