Sat Aug 26 23:06:59 PDT 2000 — Internal testing of our new web based email and application service is complete. Using ‘Twig’, you can send and receive email and manage email in folders in your home directory in a way that is compatible with other shell based email packages. It’s even got an email spell checker.

Twig also includes contact management, a to-do list, a note-pad, Usenet newsreading (read Sonic.net discussion groups from elsewhere on the net) and a calendar and appointment manager. While the tools are not as responsive as similar desktop applications like Outlook, the universal availability works great for people who use Sonic.net from many different PCs and locations.

This is our first deployment of what the industry is calling ‘application services’, so I suppose we’re now an ASP as well as an ISP. I would maintain that email, news, web hosting and the other things that we’ve been doing for years are all applications, and only those short sighted industry pundits who were recently un-shackled from their desk-tops and unleashed upon the Internet would believe that somehow deployment of a tool that’s similar to one found on the desktop moves you beyond being a simple ‘ISP’. Go figure.

Give it a try at the following URL. Let us know by posting to the sonic.help newsgroup if you find any bugs. You can find this group in the ‘News’ section within Twig. -Dane and Nathan

www.sonic.net/twig/

Thu Aug 24 16:50:35 PDT 2000 — We have deployed a new Cidera (formerly Skycache) service adaptor here for satellite reception of Usenet news. The new unit should resolve some stability problems we’ve had with the old one, and in addition, it has a third Ethernet port which is dedicated to feeding pre-loaded content for Akamai’s caches which are hosted here.

The Akamai servers locally host content here at Sonic.net for major online properties including Yahoo, CNN, Apple and many others, and having that content pre-loaded via satellite is an additional performance win. See www.sonic.net/expansion/akamai/ for details on Sonic.net’s Akamai powered partnership.

Akamai’s colocated servers serve heavy content including images and animations directly to Sonic.net customers. This means that the HTML code portion of the web page comes directly from the website you’re visiting, but special addresses in the HTML cause the images to be served locally here at Sonic.net. The images and other heavy content on the website will often be more than 80% of the total overhead, so serving this content locally is a big win.

The five Akamai machines split the workload of caching and serving up these files. Each system is built out with a full gigabyte of RAM for top performance, and caches all requests to save bandwidth and further reduce the time required to serve the page. This results not only in peak performance, but also in bandwidth savings on our upstream connections. -Dane and Kelsey

Thu Aug 24 16:46:49 PDT 2000 — Above.net has suffered cuts of multiple cross country fiber circuits in Widner, Georgia. As Above.net is the hosting location for many websites, you may notice some impact for these West coast based online properties. Supernews is hosted at Above.net, and we noticed some impact to the performance of our backup news services from them. Above.net reports now that all three circuits have been restored, and performance is returning to normal. -Dane

Thu Aug 24 14:01:45 PDT 2000 — Update: we are wrapping up the log analysis, and will send out a full list of lost emails to affected customers shortly. Meanwhile, for customers who are more technical, here’s details on the error that caused this.

We have a global procmailrc which controls delivery of email into customer mailboxes. This RC is identical across all servers, but had the ‘HOST’ tag set. We propagated a procmailrc file which contained the hostname for prop.sonic.net over to sub, ultra and buzz, and they began dropping email. This is documented in the procmailrc man page:

PROCMAILRC(5) PROCMAILRC(5)

NAME procmailrc – procmail rcfile

HOST If this is not the hostname of the machine, processing of the current rcfile will immedi- ately cease. If other rcfiles were specified on the command line, processing will continue with the next one. If all rcfiles are exhausted, the program will terminate, but will not generate an error (i.e. to the mailer it will seem that the mail has been deliv- ered).

The host tag is used if multiple machines receive a message, which might be forwarded for example during another procmail delivery, but which some machines are intended to ignore. We have no use for the host tag, and it’s been removed from our configurations. -Dane and Kelsey

Thu Aug 24 13:20:49 PDT 2000 — We have had a system failure that led to lost customer email. Beginning at 9:28:53 AM Thu Aug 24 2000 and ending at 12:15:00 PM, three out of our redundant array of five email servers were misconfigured, and silently discarded customer email. Lost messages were not bounced back to the sender, and cannot be recovered. We are shocked by this event, and have eliminated the configuration which caused this. We will be informing all senders and all recipients of each lost email message with full information including the date, time, sender and recipient of each message so that they can be re-sent.

We sincerely apologize for the error, and the inconvenience that it causes.

In addition to elimination of the configuration setup which caused this, we’re committed to deployment of a testing engine which pro-actively sends and receives email through each of the mail servers individually on an ongoing basis. By testing each and every server independently every few minutes, both locally at Sonic.net and from off site, we can stay better informed about any possible loss or delay which might affect our customers email delivery. This is similar to our existing ‘ckhosts’ and ‘ckdisk’ tools which check for availability of web, mail, ftp, Usenet news services and disk space currently.

These tools page our staff of system administrators with text messages anytime a server becomes unresponsive or unavailable, and this allows us to deliver high availability for your Internet services. We will followup here in the MOTD once this testing engine is deployed.

If you suspect that you may have lost email, please stand by for a notification. -Dane, Scott and Kelsey

Wed Aug 23 17:17:12 PDT 2000 — Just to let everyone know, we’ve increased the maximum email message size from 7MB to 15MB due to continued demand to up the size limit. We also increased the user file size limit on bolt from 24MB to 50MB. -Steve, Kelsey

Sun Aug 20 04:46:16 PDT 2000 — SOLVED: Routing loop on Cable & Wireless. CW has completed their emergency maintenance in San Francisco.Apparently the problem snowballed when they tried to replace an interface card that wasn’t supported by their router’s IOS. (Or something like that — details for their 5-hour outage are vague.) The CW T3 is up now, and I am seeing normal traffic loads. -Scott

Sun Aug 20 00:41:57 PDT 2000 — SOLVED: Packet loss and latency on UUNet. (whew!) UUNet now says this was an ATM interface that was having problems. Quote: ‘this is a known issue with Cisco ATM interfaces’. (Oh yeah? Why did it take 20 hours to fix?) Anyway, we now know the magic words to use should it happen again. Additionally, throughout the day, I’ve been writing a packet loss and latency monitoring tool that will automatically notify us of another problem like this. Called ‘See’, the tool keeps a weather eye out for problems with any of our NSP’s — a kind of minimal ‘Internet weather report.’ Please visit news:sonic.net for more information. The Cable & Wireless T3 is still down, as we are awaiting resolution of the problems with their network in San Francisco. -Scott

Sat Aug 19 23:33:00 PDT 2000 — Routing loop on Cable and Wireless. I guess I spoke too soon. I should have said ‘rest assured we won’t be bringing up the T3 until UUNet fixes their network UNLESS Cable and Wireless dies a miserable and savage death.’ Which they did: their San Francisco network just lost connectivity to the Internet. Just got off the phone with Cable & Wireless: they are conducting emergency maintenance in San Francisco. Man, when it rains, it pours… The UUNet T3 is back up (latency and all) and we will shifting back as soon as CW has completed their maintenance. -Scott

Sat Aug 19 22:33:16 PDT 2000 — Packet loss and latency on UUNet. The gentleman at UUNet tech support informed me that he saw latency. but no packet loss. I had to bring up the T3, demonstrate that there was both latency and packet loss, and then shut it down again. Each time we shut down a T3, we experience intermittent loss of connectivity to some sites while routes converge on the remaining T3’s. I apologize if you noticed a hiccup in the Internet tonight, and rest assured we won’t be bringing up the T3 until UUNet fixes their network. -Scott