Thu Dec 21 03:20:29 PST 2000 — Night Ops Complete: Our NetApps, freezer and icebox, are now configured in a cluster such that if a head unit fails, the other will seamlessly take over it’s NFS duties. Like the Alteon’s load balancing of all email, ftp and web, we now have redundancy on the NFS back-end. Thanks to Dan from NetApp we have also thoroughly tested and proven that the cluster fail-over works properly.
It should be noted that both NetApps had to be halted during the installation of the additional hardware needed to support clustering. During this time, inbound email queued locally on each mail server, but pop services were offline, along with web, ftp and shell. The total service outage extended from 12:15 AM to about 1:00 AM.
Our two new Cisco routers are on site and we’re preparing to migrate to an active-active dual router configuration using Cisco’s HSRP (similar to VSRP) protocol. Once we have finished the migration to the new Ciscos we will have full end-to-end redundancy in our core network and for all of our core services.
We also replaced our Redback’s SMS 1000 loaner with our new SMS 1800 which has a much greater capacity for expansion over our old SMS 500. The SMS terminates all PacBell and Broadlink DSL service on our network.
Lily, the T3 MUX, had it’s primary controller restored (from the last night ops.) So once again, lily is internally redundant. We also completed some routine maintenance and reorganization of our NOC and some of our core servers.
-Kelsey, Steve, Nathan, Russ, Matt, Jared, Jeff, and the guys from NetApp.