Sun Dec 14 14:32:01 PST 2003 — Oakland POP outage: The router at Oakland lost power a few minutes ago. We’re currently investigating the failure and should have an update shortly. -Nathan, Russ and Kelsey
Update at Sun Dec 14 14:44:04 PST 2003: Power has been restored, and the Oakland POP is operating normally. -Nathan and Russ
Sat Dec 13 20:46:51 PST 2003 — Focal router issues: The router at Focal in San Francisco is currently down. All dialup numbers ending in 9606 and 9811 are returning authentication failures. An employee is currently en-route to investigate the issue, and should arrive in approximately 30 minutes. As always, alternate dialup numbers are available. To locate a dialup in your area, visit:
-Nathan, Kelsey, Matt, Kavan and John
Update at Sat Dec 13 22:11:55 PST 2003: The router is now up and functional after a manual power cycle. -Nathan and John
Thu Dec 11 00:23:18 PST 2003 — AOL Mail Delivery Problems: Due to an exploited customer cgi that was being used to send SPAM to AOL customers our outbound mail servers got blacklisted by AOL. Since we became aware of this problem at 9:30PM, we were working hard to track down and disable the miscreant cgi, reroute mail flows to AOL, and contact the AOL postmaster to have the block lifted. At this time mail to AOL is still flowing through an alternate mail server. AOL has also added our servers to a whitelist and abuse notification list allowing us to respond to their customer complaints as they receive them. It may take as long as 48 hours for this whitelist to take affect, until then mail will continue to take the alternate path. -Kelsey, Nathan, Scott, John and Dane.
Sun Dec 7 20:48:54 PST 2003 — DSL outage in Santa Rosa: issue now resolved. Tonight we experienced a DSL outage affecting some Santa Rosa customers. The problem cleared itself up as we were investigating and we are currently not clear as to the cause. We will continue to monitor the situation but at this time all systems are functioning normally. We apologize for any inconvenience to those affected. – Technical support
Fri Dec 5 02:58:17 PST 2003 — Router Maintenance Update and More Kernel Upgrades: The router maintenance at Focal went more or less as planned. We encountered one issue with the new IOS release but we able to work around it. So far, so good. We hope this will restore stability in our network. We also continued our kernel upgrades on the remaining few hosts that had not already be upgraded. We did encounter some trouble here: sonic.sonic.net, or core administrative server and home to the member tools, had issues with it’s boot loader that required a couple of hours to resolve due to some other hardware related complications. Apologies for the lack of an update to the MOTD, but sonic is responsible for it’s distribution. -Kelsey and Nathan
Thu Dec 4 17:40:30 PST 2003 — Focal Router Maintenance: Tonight at Midnight we are going to working on the router at Focal. Users that are not directly connected to the focal router may notice degraded performance for the duration of the maintenance, while those that dial into the Focal pop will be isolated from the rest of our network. If everything goes as planned, the maintenance window should last for less than 10 minutes. -Nathan, John and Kelsey.
Wed Dec 3 16:25:28 PST 2003 — Upgrades and Changes to custmx: We upgrade the RAM and disks in our customer secondary MX server today to better handle the load on the box. We also enabled use of mail-abuse.org’s RBL+, a combination of the RBL, DUL, RSS and OPS, to help reduce the spam relayed by the server. These changes only affect users that run their own mail server and use our server as a backup MX handler. -Kelsey
Tue Dec 2 07:54:35 PST 2003 — Problems With the new Kernels: Our web cluster has been having stability problems with the new kernel. www.sonic.net and all of our hosted domains web sites have been having reachability issues since early this morning. We’re working to resolve the problems to assure the stability and security of the web cluster as quickly as possible. -Matt and Kelsey
Update: Tue Dec 2 22:48:52 PST 2003 — Most of the problems were solved by 9am, though some kernel issues emerged throughout the day. It appears that Kelsey and Nathan have the problem licked, but we are still monitoring the situation. -Scott
Mon Dec 1 21:50:04 PST 2003 — Kernel Upgrades: Due to a new exploit that’s been seen in the wild, we are upgrading all of the kernels on our servers. All machines which customers have access to have already been patched and rebooted. We’re continuing to work down the list of servers, eliminating any potential problems. We have run into one issue so far: the SQL server used by the SpamAssassin cluster didn’t take it’s new kernel and is currently off-line. We hope to have it back up shortly after some hands on work is performed. Until its service is restored, SpamAssassin will filter mail based on its default settings. -Kelsey and Nathan