Wed Aug 21 17:19:51 PDT 2002 — We’re observing some intermittent trouble with nas3.sr briefly caused a fast busy error tone on this dial group.
nas3.sr is 3/4 of the way into the 1003 dial group, and is generally only in use during prime time. It’s got an incorrect firmware version, which seems to be causing it to fail to accept calls. We’re moving it to the end of the dial group right now to reduce any impact, and will be changing the firmware version shortly. -Dane, John, Russ and Nathan
Wed Aug 21 04:51:27 PDT 2002 — Broadlink’s network core has been migrated to our new data center. Service to customers has been restored. The network will be under close observation for the next few hours to make sure everything is functioning properly. -Broadlink and Sonic.net
UPDATE: This morning a number of Broadlink customers have called in to report slowed connectivity and packet loss via Broadlink connections. The Broadlink operations team has been alerted and is investigating. -Broadlink and Sonic.net
UPDATE: Broadlink believes they have identified the problem and are implementing a fix for it. ETR is approximately 11:00 am. -Broadlink and Sonic.net
Wed Aug 21 14:57:13 PDT 2002 — The T1 which serves Rooftop customers in Rohnert Park ‘R’ Section is experiencing problems. We have a trouble ticket open with Pac Bell and they are investigating the problem.
Update : Wed Aug 21 17:32:16 PDT 2002 — Pac Bell traced the problem to a bad DS3 card in the Rohnert Park CO. The card was replaced and service has been restored. -Matt and Scott
Tue Aug 20 21:17:24 PDT 2002 — Dialup trouble. nas3.sr, one of the NAS boxes serving the 707-522-1003 dialup group, stopped returning IP addresses to its free pool after they had been used. This resulted in the box running out of addresses to give to clients, and hence caused 15 minutes worth of connection problems for users dialing that number. Additional addresses have been added to the pool and it is now functioning normally. To clear this trouble, we will be rebooting nas3.sr tomorrow morning at 2am. Customers connected to nas3.sr at that time will experience dropped connections.
A redundant number was available during this outage. To find dialups in your area, visit:
-Nathan and Russ
Mon Aug 19 16:36:25 PDT 2002 — Broadlink equipment move. Broadlink will be relocating their equipment from 300 B Street to our new data center on Wednesday, August 21st at 2 am. The move is expected to take about 3 hours during which time connectivity for Broadlink customers will be unavailable. If you notice problems after the scheduled move time please contact Technical Support at 707-547-3400. -Broadlink and Sonic.net
Sat Aug 17 12:44:13 PDT 2002 — One of our core routers rebooted, taking offline our link to UUNet and San Francisco briefly. This would have caused interruption of Internet some services for about 10 minutes for customers dialing our Focal dialup lines, and some congestion due to traffic re-routing to our other connections in the core.
Sorry for the late MOTD – it was overlooked on Saturday. This MOTD was entered Mon Aug 19 22:46:07 PDT 2002. -Dane
Fri Aug 16 09:06:29 PDT 2002 — Spamassassin server reboot. The server that evaluates and tags spam email was rebooted this morning because it stopped tagging email. This resulted in messages not getting tagged as spam. The server is now functioning properly.
To learn more about Spamassassin and Sonic.net’s spam defenses read the Spamassassin FAQ at:
www.sonic.net/support/faq/advanced/spamassassin.shtml
Update: The user database server on the Spamassassin server came up in a weird state. This has been corrected, and the server has been reconfigured to prevent this in the future. -Scott -Matt
Fri Aug 16 19:45:50 PDT 2002 — Fast busies on 522-1003. One of our Total Control modem banks stopped taking calls this evening at 6pm. Rebooting the equipment fixed the problem. -Matt, Russ and Scott
Tue Aug 13 18:18:35 PDT 2002 — Intermittent mail delays. We’ve had difficulties with intermittent delay of local email, resulting in suboptimal queue sizes. This has become quite noticeable the past few days, especially today. This was a main agenda topic at today’s operations meeting, where we developed a multi-pronged plan of attack to get the situation under control. Current queue sizes for mail servers in our pool are low, and we have confirmed that the majority of these messages are store-and-forward to networks that have not yet collected. We will post updates as we work further to improve email, our most important application. -Scott, Dane, Kelsey, Russ, Matt
Mon Aug 12 01:41:32 PDT 2002 — The circuit that is used for our Covad DSL uplink has been unstable since 1:17AM. It’s gone through over 2000 carrier transitions in the past half hour! We our en route to the data center to investigate the cause the flapping. In the meantime we have administratively shutdown the interface to prevent routing instability in our network. We hope to have service restored to our Covad DSL customers shortly. -Kelsey
Update: The circuit appears to have stabilized; all service has been restored. We’ll keep a close eye on the circuit for the next few hours.