Tue Aug 27 12:35:46 PDT 2002 — Rooftop Customers in Rohnert Park ‘R’ Section have lost connectivity. Pac Bell has a fiber hut which has been experiencing problems over the last week. We have opened a ticket and are waiting for a response. -Matt
Tue Aug 27 14:25:22 PDT 2002 — Pac Bell has completed repair on the circuit and service has been restored to customers.
Tue Aug 27 17:03:31 PDT 2002 — On Thursday morning at 1am we will be performing various maintenance tasks on our network. We will be updating our BGP configuration to allow us to better serve BGP to colocated and HiCap customers. This reconfiguration will not affect service to customers.
We will also be upgrading the IOS on Mega, a router which serves customer T1’s and T3’s. We expect no more than 5 minutes of service interruption while the router reboots. The new code has numerous enhancements including an OSPF update which fixes a routing bug. -Matt, Nathan and Zeke
Sat Aug 24 11:37:15 PDT 2002 — Rooftop Customers in Rohnert Park ‘R’ Section have lost connectivity due to a Pac Bell T1 failure. Pac Bell has been notified of the failure and they are working on the problem.
Update: Sat Aug 24 17:34:45 PDT 2002, Pac Bell was able to determine that the problem was with a DSX card in Rohnert Park. They have reset the card and the circuit is functioning again. This appears to be a chronic issue with this card and they have promised to replace it if it happens again.
-Matt and Dane
Fri Aug 23 17:10:12 PDT 2002 — Broadlink is currently experiencing problems in their network. Some of their WDSL connected customers are offline at this time and they are working hard to find the common theme so they can restore service to these customers. -Kelsey, Chris and Broadlink
Update: Broadlink reports that service is restored to all customers.
Fri Aug 23 15:58:34 PDT 2002 — New ZDI tool. We’ve reworked our Zero Disconnect Initiative tool to retrieve information directly from our RADIUS server accounting logs, rather than using a database. We expect improved detection of dialup member disconnect problems, for which we take a proactive stance. -Scott
Thu Aug 22 09:50:02 PDT 2002 — SpamAssassin issues: We are seeing around 500 SA client connections fail to our dedicated SA server a day. No email messages are lost when this occurs but they are not filtered by SA. We have been investigating the cause of these errors for a few days but haven’t found anything wrong with either the SA client or server. We are continuing to review the code and hope to have the problem resolved shortly. Once we’ve fixed the bug we will submit patches to the SA CVS committers. -Kelsey and Nathan
Thu Aug 22 17:49:56 PDT 2002 — Night Ops: We are going to load new IOS code on our Cisco edge router at our San Francisco POP. Cisco has identified the bug that caused the router to reboot last weekend and has provided us with new code. This router terminates our Focal dial-up circuits as well as our uu.net T3. It should take no more than a few minutes for the router to reload and return to service. We will also be taking this opportunity to do some network maintenance at the Focal POP. Focal dialup numbers xxx-9811 and xxx-9606 may be unavailable briefly and users currently online may be kicked off. We will start the night ops at 2:00AM Friday morning. -Nathan, Russ, Matt and Kelsey
Thu Aug 22 17:31:06 PDT 2002 — SpamAssassin issues: We have patched the client to retry connections a few times before passing the unfiltered email on for local delivery. This should resolve the issues with periodic unfiltered messages reaching user’s email boxes. For more information on SpamAssassin, and how it can rid your email box of SPAM, please see news://news.sonic.net/sonic.antispam -Nathan and Kelsey
Thu Aug 22 13:03:47 PDT 2002 — Our radius authentication servers experienced a brief overload when an apparent bug got tickled by with a routine configuration update. We removed the conflicting configuration and the problem has been resolved. The event may have resulted in slow and possibly failed authentication. We’ll be reviewing the source code and will patch the servers to prevent this from happening again. -Kelsey, Nathan and Russ
Wed Aug 21 11:26:17 PDT 2002 — Broadlink customers are still seeing performance problems in the wake of the Broadlink migration last night. Some problems have been addressed, but customers are still seeing reduced speeds and packet loss. Broadlink is hard at work trying to restore normal service and we will announce it here as soon as they have resolved the issue. -Broadlink and Sonic.net
UPDATE: Broadlink has finished making their adjustments and normal service has been restored to all Broadlink customers. -Broadlink and Sonic.net