Tonight at midnight we will be performing potentially invasive maintenance on one of our ATM switches. We will be reloading the primary route processor in the ATM switch and it should automatically fail over to the backup. Impact should be only a few seconds as the failover occurs, though it may be up to 10 minutes if problems are encountered. The ATM switch in question serves DSL and Business-T customers in the Bay Area, Sacramento and Fresno areas.
-Jared and Nathan
Update: The failover to the backup route processor did not go smoothly, so we were forced to reboot the ATM switch chassis. This resulted in approximately 10 minutes of connectivity interruption for customers connected through this ATM switch. We apologize for the interruption. We are monitoring traffic levels across the ATM switch and they appear to be moving back to normal levels. We will continue to monitor the switch closely.
Yup, it just went dead here in Petaluma at about 12:30am. The failover didn’t seem to work – Still dead 10 minutes later. :). Perhaps a hint that I shouldn’t be working this late anyway. 😉
Over 25 minutes now and I really do have to work. 🙁
I’m back up now, but I had to power-cycle my DSL router before it would come back up (after almost an hour after the outage). 🙁
Sorry about that Mark, as you surmised, the operation did not go as smoothly as it could have. I did take a look at your DSL line after making sure the switch reboot took care of our problem, and I saw that I could ATM ping your DSL line, but that no new DHCP requests had come in from your line. Our dynamic IP DSL clients should be renewing their DHCP leases every 5-10 minutes, but I have seen some SOHO routers be stubborn about this, meaning that in order to get them to re-lease an IP, they need to be rebooted.
I too had to power-cycle my DSL modem this morning, but no worries-it’s working now.
Thanks,
Adam
To follow up: we are aware that some DHCP clients do not obey the short lease time that we offer them, thus making them take longer to get back online in the case of a systems failure/invasive maintenance. We have been looking into changes we can make to our DHCP configuration to alleviate the problem that is described here.
In the meantime, if your dynamic IP DSL connection stops working, it’s never a bad idea to reboot the DSL modem and router, just in case.