Author: admin

Tue Apr 15 21:35:30 PDT 2003 — Router reboot. The router which serves 522-1001 and 522-1002 rebooted this evening at 9:27pm. It was offline for about 3 minutes while it loaded. During this time, customers connected to those dial-up numbers lost connectivity. Service has been restored and we are looking for an explanation for the event. -Matt and Nathan

Mon Apr 14 20:55:50 PDT 2003 — Busies on 522-1001 and 1002: One of the USR TC hubs that handles these lines is having trouble on three of it’s PRIs. (Each PRI handles 23 modems.) And is returning occasional fast-busies despite excess capacity. We’re working with ATG to see if they can identify the fault from their side. We already marked these lines out-of-service but apparently calls are still getting routed to them. -Kelsey and Russ

Sun Apr 13 02:02:35 PDT 2003 — Night Ops Complete: All work went as planned. We have replaced the AD3 with one of the ones from the new AD3 cluster earmarked for the new mail cluster. So far it appears to be functioning perfectly. Apparently the old AD3 was suffering from a hardware failure; one which none of the Nortel support reps we’ve talked to have ever seen or heard of. We will be keeping a close eye on the new box; we were able to ‘fix’ the old AD3 a number of times only to have it fail again one week or even a month later. Recent experience aside, the Alteon AD3 has been a very reliable and trusted component of our network. -Kelsey and Nathan

Sat Apr 12 13:16:03 PDT 2003 — Night Ops: Sunday, at 12:00AM, we are going to replace the AD3 that is currently having trouble with another AD3 that is currently deployed in an Active-Standby cluster with a partner for the new mail server cluster. We’ve been reluctant to swap hardware until we had discussed the situation with Nortel who now owns the Alteon product line. Any downtime experienced during the swap to the new AD3 should be less then the periodic outages that we are currently seeing.

We are also taking this opportunity to reboot our Netapp NFS cluster in order to fix a minor annoyance that needs a reboot to clear. They should be down for no more than a few minutes. At the same time the NetApps are being rebooted we will also be swapping the shell server’s disk back into its own chassis to free up the spare server for some other tasks. -Kelsey and Nathan

Fri Apr 11 03:42:22 PDT 2003 — Trouble with the Alteon: The AD3 switch that currently handles load balancing our web and mail services has stopped co-operating with the Cisco switches that are on either side of it. The resulting problem, like before, is intermittent packet-loss which may result in poor performance while trying to send or receive mail or accessing a locally hosted web page. We’ve already replaced all of the hardware on all sides of the AD3 in attempts to resolve this problem but we’ve only succeeded in ‘fixing’ it for a week or two at a time before the problem shows up again. We are considering all options available; resolving this chronic problem in both the short and long term is our highest priority. -Kelsey and Nathan.

Fri Apr 11 14:30:05 PDT 2003 — Authentication problems, xxx-9811 pool. Shortly after 1:00pm today, we stopped authenticating successfully for clients dialing our xxx-xxx-9811 numbers. Support was closed for our weekly meeting, and a (soon to be corrected) fault in our monitoring system prevented us from fixing it as quickly as we could have. Eli, Scott, Russ.

Thu Apr 10 19:59:38 PDT 2003 — Our SpamAssassin cluster experienced some load related failures this afternoon while we had taken one of the three servers offline in preparation to add the fourth dedicated SpamAssassin server tomorrow morning. The addition of the fourth server will restore redundancy to the cluster and should help limit the occasional email that passes through the cluster unchecked. -Kelsey and Kevan.

Wed Apr 9 08:00:00 PDT 2003 — At 8am this morning, our LATA-9 (Stockton) DSL concentrator lost connectivity to the Internet for approximately 20 minutes. It turns out the Stockton facility deployed a new layer 3 switch, which caused difficulties with our router. Only LATA-9 DSL customers were affected. -Scott, Nathan, and Dane

Mon Mar 31 13:05:55 PST 2003 — Site Wide Sendmail Upgrades: We are busy rebuilding sendmail for all of our servers to patch the second severe sendmail exploit released this month. Most of our internal servers have been updated as well as bolt. Some users may have experienced some trouble sending mail from bolt while the new binary was being installed and the configuration was getting properly tweaked. The updates will be completed within an hour on the rest of our servers. -Kelsey and Russ