Fri Dec 27 09:29:24 PST 2002 — Our UU.net T3 is currently off-line. We are in contact with their NOC and are working with them to bring it back on line as soon as possible. At this time we do not have an ETA. We will update the MOTD as soon as we have more information. -Kelsey and Nathan

Update: The UU.net T3 came back online just as we started to do testing with UU.net’s NOC. We will both be keeping a close eye on the circuit for the next few days to see if any other errors occur. We don’t believe that any customers should have noticed this outage as our other circuits were able to sustain the additional traffic load. -Kelsey and Nathan

Wed Dec 25 20:35:49 PST 2002 — All circuits are busy on 9811 dial-up numbers. We are experiencing “All circuits are busy” on our focal hunt group. We are currently investigating the problem. In the mean time, please check the POP finder at www.sonic.net/cgi-bin/pops.pl for alternate dial-up numbers. -Matt, Eli and Kelsey

Update, Wed Dec 25 20:53:54 PST 2002 — We have reboot the NAS gear and are now accepting calls. We will monitor the hunt group over the next few hours but don’t expect to see more problems.

Sat Dec 21 10:32:23 PST 2002 — SSL server problem. This morning, the SSL server exhibited the same behavior we’ve seen in the past, where it accepts a connection but then stalls. Investigation reveals that the daemon refresher event failed to kill the old httpd process, so the new process couldn’t run. This has been solved by making the “kill” program more aggressive. Hopefully we won’t have this problem again. -Scott and Nathan R.

Sat Dec 21 15:15:41 PST 2002 — sms reboot. The router which serves Pac Bell DSL customers rebooted about 5 minutes ago. We will be keeping a close eye on it over the next few hours, but don’t anticipate any more problems. All service has been restored. -Matt, Eli and Kavan

Fri Dec 20 23:11:18 PST 2002 — We have resolved the improper routing of mail addressed to our users at redwood.net, zapcom.net, and community.net, three ISPs that we have acquired in the past. At this time, only users who where migrated from these ISPs can receive mail at their respective domains. This should resolve the apparent mis-delivery of some mail as well as help reduce the volume of SPAM that some users receive. The remaining customer domains that share this problem will be fixed shortly. -Kelsey and Dane.

Wed Dec 18 09:34:34 PST 2002 — Broadlink tower outage. Broadlink has determined that the backhaul radio at their Barham tower has failed. Broadlink says that, worst case, they should have all known hardware failures at the tower fixed before noon today. Once the tower is restored, they will verify that service has been restored to each and every customer, and schedule repairs if necessary.

According to PG&E line power to the Barham tower won’t be restored until sometime after 12/20/02. When power is restored the switch from the generator over to wired power may cause a brief outage (5-10 minutes). -Jason and Tim at Broadlink, Scott and Dane at Sonic.net

Wed Dec 18 18:21:33 PST 2002 — Mail server trouble. Two or more of our mail servers refused mail for short periods of time today with ‘User Unknown’ errors for at least some of the email they received. Although we were unable to make the new DB passwd lookups fail during testing, we believe that this failure is related to our recent upgrades. We’ve re-enabled NIS as a fall back user lookup method in case DB lookups fail again. In the meantime we will further analyze the failures so we can prevent them from occurring again again. -Kelsey, Eli, Russ and Scott.

Mon Dec 16 12:21:18 PST 2002 — Power restored. At approximately 8:30am, power was restored to Sonic.net’s Apollo headquarters. -Scott

Mon Dec 16 03:40:11 PST 2002 — Sonic.net on generator power. For the first time ever, Sonic.net is experiencing an extended power event at our headquarters. Everything is online and performing as expected, and no manual intervention is needed.

I’ve visited the office to make sure that all is well, and checked the diesel fuel level – we’re still between 90% and 100% fuel after running on generator for more than two hours. Our fuel and load projections indicate that we’ve got enough diesel to run for days – and of course, we’ll be topping up the tank in the AM Monday. Generator load is currently less than 10% of the max load of 750 kilowatts (3/4 of a megawatt, enough power for a small town of about 750 homes); this generator is sized for the ten year plan.

No services have been interrupted. Kudos to McClure Electric, JLC Construction, Stuart & Stevens gensets and Honeywell monitoring; and of course, Detroit Diesel.

-Dane (proudly)

Mon Dec 16 14:51:19 PST 2002 — Miscellaneous Upgrades. We have completed the migration away from YP (NIS) to db files for passwd and group lookups on all of the mail and SpamAssassin servers. This eliminates a number of problems and should also improve performance on these servers, especially when under high load.

We have upgraded our domain email alias membertool to correct a problem wherein users were able to remove their ‘catch-all’ alias causing mail to be incorrectly delivered to other sonic.net mailboxes. Over the next two weeks we are going to work to correct any domains that do not currently have catch-all aliases set up. This should prevent the delivery of mail to the wrong entities, as well as help reduce the amount of SPAM that some users receive. -Kelsey, Nathan, Scott, John, Russ, and Chris B.