Fri Aug 23 17:10:12 PDT 2002 — Broadlink is currently experiencing problems in their network. Some of their WDSL connected customers are offline at this time and they are working hard to find the common theme so they can restore service to these customers. -Kelsey, Chris and Broadlink
Update: Broadlink reports that service is restored to all customers.
Fri Aug 23 15:58:34 PDT 2002 — New ZDI tool. We’ve reworked our Zero Disconnect Initiative tool to retrieve information directly from our RADIUS server accounting logs, rather than using a database. We expect improved detection of dialup member disconnect problems, for which we take a proactive stance. -Scott
Thu Aug 22 09:50:02 PDT 2002 — SpamAssassin issues: We are seeing around 500 SA client connections fail to our dedicated SA server a day. No email messages are lost when this occurs but they are not filtered by SA. We have been investigating the cause of these errors for a few days but haven’t found anything wrong with either the SA client or server. We are continuing to review the code and hope to have the problem resolved shortly. Once we’ve fixed the bug we will submit patches to the SA CVS committers. -Kelsey and Nathan
Thu Aug 22 17:49:56 PDT 2002 — Night Ops: We are going to load new IOS code on our Cisco edge router at our San Francisco POP. Cisco has identified the bug that caused the router to reboot last weekend and has provided us with new code. This router terminates our Focal dial-up circuits as well as our uu.net T3. It should take no more than a few minutes for the router to reload and return to service. We will also be taking this opportunity to do some network maintenance at the Focal POP. Focal dialup numbers xxx-9811 and xxx-9606 may be unavailable briefly and users currently online may be kicked off. We will start the night ops at 2:00AM Friday morning. -Nathan, Russ, Matt and Kelsey
Thu Aug 22 17:31:06 PDT 2002 — SpamAssassin issues: We have patched the client to retry connections a few times before passing the unfiltered email on for local delivery. This should resolve the issues with periodic unfiltered messages reaching user’s email boxes. For more information on SpamAssassin, and how it can rid your email box of SPAM, please see news://news.sonic.net/sonic.antispam -Nathan and Kelsey
Thu Aug 22 13:03:47 PDT 2002 — Our radius authentication servers experienced a brief overload when an apparent bug got tickled by with a routine configuration update. We removed the conflicting configuration and the problem has been resolved. The event may have resulted in slow and possibly failed authentication. We’ll be reviewing the source code and will patch the servers to prevent this from happening again. -Kelsey, Nathan and Russ
Wed Aug 21 11:26:17 PDT 2002 — Broadlink customers are still seeing performance problems in the wake of the Broadlink migration last night. Some problems have been addressed, but customers are still seeing reduced speeds and packet loss. Broadlink is hard at work trying to restore normal service and we will announce it here as soon as they have resolved the issue. -Broadlink and Sonic.net
UPDATE: Broadlink has finished making their adjustments and normal service has been restored to all Broadlink customers. -Broadlink and Sonic.net
Wed Aug 21 17:19:51 PDT 2002 — We’re observing some intermittent trouble with nas3.sr briefly caused a fast busy error tone on this dial group.
nas3.sr is 3/4 of the way into the 1003 dial group, and is generally only in use during prime time. It’s got an incorrect firmware version, which seems to be causing it to fail to accept calls. We’re moving it to the end of the dial group right now to reduce any impact, and will be changing the firmware version shortly. -Dane, John, Russ and Nathan
Wed Aug 21 04:51:27 PDT 2002 — Broadlink’s network core has been migrated to our new data center. Service to customers has been restored. The network will be under close observation for the next few hours to make sure everything is functioning properly. -Broadlink and Sonic.net
UPDATE: This morning a number of Broadlink customers have called in to report slowed connectivity and packet loss via Broadlink connections. The Broadlink operations team has been alerted and is investigating. -Broadlink and Sonic.net
UPDATE: Broadlink believes they have identified the problem and are implementing a fix for it. ETR is approximately 11:00 am. -Broadlink and Sonic.net
Wed Aug 21 14:57:13 PDT 2002 — The T1 which serves Rooftop customers in Rohnert Park ‘R’ Section is experiencing problems. We have a trouble ticket open with Pac Bell and they are investigating the problem.
Update : Wed Aug 21 17:32:16 PDT 2002 — Pac Bell traced the problem to a bad DS3 card in the Rohnert Park CO. The card was replaced and service has been restored. -Matt and Scott