Tue Aug 20 21:17:24 PDT 2002 — Dialup trouble. nas3.sr, one of the NAS boxes serving the 707-522-1003 dialup group, stopped returning IP addresses to its free pool after they had been used. This resulted in the box running out of addresses to give to clients, and hence caused 15 minutes worth of connection problems for users dialing that number. Additional addresses have been added to the pool and it is now functioning normally. To clear this trouble, we will be rebooting nas3.sr tomorrow morning at 2am. Customers connected to nas3.sr at that time will experience dropped connections.

A redundant number was available during this outage. To find dialups in your area, visit:

www.sonic.net/cgi-bin/pops.pl

-Nathan and Russ

Mon Aug 19 16:36:25 PDT 2002 — Broadlink equipment move. Broadlink will be relocating their equipment from 300 B Street to our new data center on Wednesday, August 21st at 2 am. The move is expected to take about 3 hours during which time connectivity for Broadlink customers will be unavailable. If you notice problems after the scheduled move time please contact Technical Support at 707-547-3400. -Broadlink and Sonic.net

Sat Aug 17 12:44:13 PDT 2002 — One of our core routers rebooted, taking offline our link to UUNet and San Francisco briefly. This would have caused interruption of Internet some services for about 10 minutes for customers dialing our Focal dialup lines, and some congestion due to traffic re-routing to our other connections in the core.

Sorry for the late MOTD – it was overlooked on Saturday. This MOTD was entered Mon Aug 19 22:46:07 PDT 2002. -Dane

Fri Aug 16 09:06:29 PDT 2002 — Spamassassin server reboot. The server that evaluates and tags spam email was rebooted this morning because it stopped tagging email. This resulted in messages not getting tagged as spam. The server is now functioning properly.

To learn more about Spamassassin and Sonic.net’s spam defenses read the Spamassassin FAQ at:

www.sonic.net/support/faq/advanced/spamassassin.shtml

Update: The user database server on the Spamassassin server came up in a weird state. This has been corrected, and the server has been reconfigured to prevent this in the future. -Scott -Matt

Fri Aug 16 19:45:50 PDT 2002 — Fast busies on 522-1003. One of our Total Control modem banks stopped taking calls this evening at 6pm. Rebooting the equipment fixed the problem. -Matt, Russ and Scott

Tue Aug 13 18:18:35 PDT 2002 — Intermittent mail delays. We’ve had difficulties with intermittent delay of local email, resulting in suboptimal queue sizes. This has become quite noticeable the past few days, especially today. This was a main agenda topic at today’s operations meeting, where we developed a multi-pronged plan of attack to get the situation under control. Current queue sizes for mail servers in our pool are low, and we have confirmed that the majority of these messages are store-and-forward to networks that have not yet collected. We will post updates as we work further to improve email, our most important application. -Scott, Dane, Kelsey, Russ, Matt

Mon Aug 12 01:41:32 PDT 2002 — The circuit that is used for our Covad DSL uplink has been unstable since 1:17AM. It’s gone through over 2000 carrier transitions in the past half hour! We our en route to the data center to investigate the cause the flapping. In the meantime we have administratively shutdown the interface to prevent routing instability in our network. We hope to have service restored to our Covad DSL customers shortly. -Kelsey

Update: The circuit appears to have stabilized; all service has been restored. We’ll keep a close eye on the circuit for the next few hours.

Thu Aug 8 13:45:35 PDT 2002 — CW outage. At about 1:30pm CW dropped our T3 to them. It is now back up again, and engineers are monitoring it for further signs of trouble. -Scott and Nathan

Thu Aug 8 13:34:35 PDT 2002 — CW does it AGAIN. Cable and Wireless has once again killed one of our T3’s. We have escalated the ticket and are working with their NOC to get the circuit up as soon as possible. Meanwhile, our UUNet T3 is fine, but is groaning under the load. -Scott

Wed Aug 7 09:29:29 PDT 2002 — This morning our authentication servers experienced problems with authenticating users. After spending some time trouble shooting the issue we were able to get it back on line. This caused dialup customers to get an invalid password message when trying to log in. We apologize for the inconvenience this may have caused. – Ops Team