Sat Nov 9 13:06:43 PST 2002 — Miscellaneous Broadlink Problems. Broadlink has had some problems due to the windstorm. They currently have one tower completely down, which they are working to restore. Additionally, Broadlink customers may be impacted due to wind misaligning their antennas. If your Broadlink service has degraded since the windstorm, please email support@sonic.net, or call tech support at 547-3400. -Scott and Dane

Fri Nov 8 14:09:32 PST 2002 — A brief denial of service attack against one of our colocated customers caused some intermittent performance and delays.

We are in communication with the customer, and it’s possible that their machine may be compromised. If the issue occurs again, we can place bandwidth limits on the port to eliminate the possibility that this would impact our uplinks and other customers. -Dane, Kelsey, Matt and Kevan

Thu Nov 7 09:49:26 PST 2002 — Busies on 707-522-1003. We are currently experiencing intermittent busy signals on our 1003 dial-up group and are opening a trouble ticket with Pac Bell. Please remember that we have backup numbers available at: www.sonic.net/cgi-bin/pops.pl -Matt

Tue Nov 5 10:31:56 PST 2002 — ATM OC-3 upgrade. DSL and some Frame Relay customers will have service briefly interrupted during an upgrade of our ATM circuit from a T3 to an OC-3. This upgrade will increase capacity on our existing link, which is currently experiencing congestion during peak utilization periods.

The migration of customer connections to the new circuit is scheduled to occur between 2am and 5am on Wednesday November 13th.

For daily statistics on the current circuit, see the following URL. Note that the average utilization outbound is about 32 megabits, and it’s peaking at it’s max capacity of 36 megabits.

makeashorterlink.com/?E48B23A52

The new OC-3 circuit is three times as fast as the current T3 and has a raw speed equivalent to 84 T1 circuits (ATM protocol overhead reduces this somewhat). Customers will experience lower latency and higher speeds during peak utilization times subsequent to this upgrade. -Dane

Tue Nov 12 13:26:28 PST 2002: UPDATE. This upgrade has been rescheduled for next Wednesday, as SBC was not ready for the migration as planned. Any orders which were held pending the upgrade are being processed today.

Wed Nov 20 04:14:53 PST 2002: UPDATE. OC3 migration complete. We have successfully migrated off the ATM DS3 and on to the new ATM OC3. This involved moving all FRATM and DSL customers in blocks from one circuit to the other over a period of approximately 2 hours and 15 minutes. This new capacity will eliminate the congestion we were experiencing during peak times. During the migration we also moved an external CSU/DSU chassis which serves Covad, frame-relay, peering and customer T1’s. Pac Bell projected a 3 hour migration, but thanks to Augie at ASI, things went more quickly. -Matt, Zeke and Scott

Tue Nov 5 02:39:01 PST 2002 — Issues with SpamAssassin: There were reports yesterday of email reaching customers’ inboxes without being filtered by our anti-spam tools. We were unable to correlate any of the reported failures to any problem on our servers. We were also misled by the belief that the failures only occurred during a short window when some routine maintenance was being performed on the load balanced spam processing servers. Some users brought additional information to our attention in news://news.sonic.net/sonic.antispam and we were able to correlate their reported failures with some issues on our end. This intermittent failure only affected customers that were using the old ‘file based’ SpamAssassin and not the new web-based tools. The problem has been resolved and will be investigated further in the morning. -Kelsey

Mon Nov 4 15:14:10 PST 2002 — Squirrelmail. For a few weeks we have had select users beta-testing a new web mail client for us, and reports have been overwhelmingly positive. Today, this new program, Squirrelmail, is replacing TWIG as our default web mail application. You can reach it through the web mail link on the main sonic.net home page, or through www.sonic.net/mail directly. Please note that TWIG will still be available directly through www.sonic.net/twig

Note: Your contacts can be easily imported from TWIG with a single click in the addresses section. Just click the “Import TWIG Contact list” link on the bottom. -Kevan

Fri Nov 1 16:00:38 PST 2002 — Ultra, one of our 5 load balanced mail servers entered an unusual failure mode where it could no longer resolve the IP of our outbound SMTP server cluster. 797 outbound email messages were returned to their local senders before we were made aware of the problem and removed ultra from the mail server pool. This only seems to have affected outbound email delivery from ultra. We apologize for the problem and are working to make sure that it does not occur again. -Kelsey, Eli and Scott

Fri Nov 1 14:42:28 PST 2002 — On Monday night at midnight we will be making a full test of our power generation facilities. While we certainly don’t have any reason to expect an interruption of power during this transition, it is possible. We plan to run the ISP on diesel for about 30 minutes during this test.

This will be the first full load test of the generator. Two previous partial load tests and periodic no load tests have gone well. Once the system is proven at full load, we will be doing periodic full load runs at least once per month during the daytime.

Sonic.net’s power generation system is a 24 liter V-12 twin turbocharged Detroit Diesel, which generates 1024 horsepower and 750,000 watts of power. This is enough electricity to power a small town of about 750 homes – or, one rather large ISP. A huge Leibert UPS array keeps us online during generator startup.

Tue Nov 5 10:32:16 PST 2002: Update; due to a scheduling difficulty, this test has been delayed until Tuesday at midnight.

Wed Nov 6 00:35:06 PST 2002: Update; as I write this, Sonic.net is running entirely on diesel power. The full transition test went smoothly, and all power generation and transfer systems operated as expected.

Our power generation plant can keep Sonic.net running indefinitely in the event of a utility failure. We have enough diesel on site currently to run for a week, and a fueling truck is scheduled to visit as often as we need. -Dane

Fri Nov 1 14:23:47 PST 2002 — BroadLink had a scheduled power outage at one of their tower sites this morning, but the UPS system failed. They replaced the equipment quickly to get customers back online. -Dane

Update, Fri Nov 1 17:32:08 PST 2002: A second outage occurred, and has been resolved. We expect at least one more once PG&E competes their work. As it’s both informative and funny, I’ll include an excerpt from the internal Sonic.net/BroadLink staff discussion list that explains the trouble. The following was written by BroadLink’s wonderful Jason Kane:

Regarding what happened:

As noted in the previous message PG&E was putting up a new power poll across the street from the tower site. As a result everyone in that area lost power for the day. It’s my understanding that wire line power will be restored in a few hours.

We originally believed that the scheduled power outage would not effect our customers since we have battery backup and a generator to re-charge it. The UPS failed immediately and the tower went dark. That was this morning.

To fix the problem we replaced the UPS with a mostly-charged unit, gassed up the generator, plugged our hardware into the UPS and plugged the UPS into the generator. Everything came back up and we figured our only problem was making sure the generator had plenty of gas. But the universe decided that today would be a good day for tweaking with the otherwise idyllic lives of BL and it’s faithful customers. As you may be aware, the tower took a nose-dive about twenty minutes ago. We had Tim nearby so he checked it out. The assumption was (of course) that the generator ran out of gas. But low and behold the generator was still cranking along without a hitch (a questionable metaphor but you get the idea). The UPS on the other hand was in a world of it’s own. And in that world, restarting every few seconds is some sort of imperative.

When we unplug the UPS from the generator it emerges from this malady and runs the tower off battery. But as every little gelfling knows you can’t run on batteries forever, even if you’re a pink bunny. So we rework the setup and plug everything into the generator. Fingers crossed the switches are flipped and the information age continues.

It is my belief that the power from the generator was not smooth/clean enough for the UPS. While it’s rather strange to have a UPS that’s more picky about clean juice than the switches, radios, management units and imported dancing hula girl lamps that are stuffed into the tower we’re forced by irrevocable circumstance to continue living without an adequate answer to such questions.

Here’s the basic sequence of events:

8:20am – PG&E disconnects power to tower 8:23am – broadlink battery backup system fails … outage 9:33am – battery system replaced, generator added 2:43pm – power supplied by generator kills replacement battery system … outage 3:33pm – tower rewired to run directly off generator, service restored

PG&E is supposed to finish their work today so we can rewire to run off primary power once again. We’ll test the battery system and replace it if needed to prevent a similar problem from recurring. We also now know that our generator can’t be used to re-charge our battery backups.

-Jason (BroadLink)

I hope that you found this as amusing as I did. -Dane

Thu Oct 31 13:14:49 PST 2002 — SpamAssassin and Graymail updates: We’ve just completed upgrading SpamAssassin to the latest stable release and added a new feature to the GUI. Users can now optionally choose to have our servers dump blacklisted senders so they’ll neither end up in your inbox nor graymail. Please keep in mind that it does no good to blacklist an address that isn’t going to be reused by the sender; most spam has uniquely generated From addresses. We have received overwhelmingly positive feedback for SpamAssassin and over 30% of our members have turned it on. SpamAssassin is now enabled by default for all new accounts. We strongly recommend that anyone who hasn’t enabled SpamAssassin and graymail do so now. You can enable it in the member tools at sonic.sonic.net/membertools/spamcan -Kelsey and Chris B.