Author: admin

Mon Aug 28 19:14:06 PDT 2000 — We’re working on a few bugs in twig, our new web based email, newsreading and productivity package. Some users have reported twig has moved their email from the inbox into a mailbox ‘mbox’, and then they can’t retrieve the same email with POP. We haven’t been able to duplicate this trouble internally yet, but we’re looking at it! -Dane and Scott

Mon Aug 28 18:54:45 PDT 2000 — We will be upgrading interfaces in the core router tonight around 1am. This should not result in any interruption of services, as all the interfaces are hot swap. We will be taking the Cable & Wireless T3 offline for a little while during this upgrade, but we will leave the UUNet T3 online, so this should not affect customer Internet access. The upgrade includes testing of a new UUNet T3, plus the installation of a new channelized T3 card for serving customer T1 circuits.

As a side note, our new T1 service offering will allow us to provision true point to point full T1 circuits to customers in and near Santa Rosa for a total delivered cost of $999/mo. Get them while they’re hot – the channelized T3 interface will house 28 customer T1 circuits, so if you want one, order early. If you’re interested in ordering, email sales@sonic.net and let us know. -Dane and Scott

Sun Aug 27 21:16:46 PDT 2000 — Problems with Guerneville and Occidental -9811 numbers. We’ve had a few lines in West Sonoma County come up with a recording ‘the service isn’t available on this line’ instead of carrier. We suspect a call routing problem to our Focal PRI’s here in Santa Rosa — either that, or the calls aren’t rolling from the -9811 numbers onto our two new Focal PRI’s.

In any event, the problem has cleared now, and we have submitted a request for ‘reason for failure’ from Focal. We’ll also be double-checking that all our Focal PRI’s are available to service -9811 calls. Finally, please note that this did not effect any number that _doesn’t_ end with -9811. -Scott

Sat Aug 26 23:06:59 PDT 2000 — Internal testing of our new web based email and application service is complete. Using ‘Twig’, you can send and receive email and manage email in folders in your home directory in a way that is compatible with other shell based email packages. It’s even got an email spell checker.

Twig also includes contact management, a to-do list, a note-pad, Usenet newsreading (read Sonic.net discussion groups from elsewhere on the net) and a calendar and appointment manager. While the tools are not as responsive as similar desktop applications like Outlook, the universal availability works great for people who use Sonic.net from many different PCs and locations.

This is our first deployment of what the industry is calling ‘application services’, so I suppose we’re now an ASP as well as an ISP. I would maintain that email, news, web hosting and the other things that we’ve been doing for years are all applications, and only those short sighted industry pundits who were recently un-shackled from their desk-tops and unleashed upon the Internet would believe that somehow deployment of a tool that’s similar to one found on the desktop moves you beyond being a simple ‘ISP’. Go figure.

Give it a try at the following URL. Let us know by posting to the sonic.help newsgroup if you find any bugs. You can find this group in the ‘News’ section within Twig. -Dane and Nathan

www.sonic.net/twig/

Thu Aug 24 16:50:35 PDT 2000 — We have deployed a new Cidera (formerly Skycache) service adaptor here for satellite reception of Usenet news. The new unit should resolve some stability problems we’ve had with the old one, and in addition, it has a third Ethernet port which is dedicated to feeding pre-loaded content for Akamai’s caches which are hosted here.

The Akamai servers locally host content here at Sonic.net for major online properties including Yahoo, CNN, Apple and many others, and having that content pre-loaded via satellite is an additional performance win. See www.sonic.net/expansion/akamai/ for details on Sonic.net’s Akamai powered partnership.

Akamai’s colocated servers serve heavy content including images and animations directly to Sonic.net customers. This means that the HTML code portion of the web page comes directly from the website you’re visiting, but special addresses in the HTML cause the images to be served locally here at Sonic.net. The images and other heavy content on the website will often be more than 80% of the total overhead, so serving this content locally is a big win.

The five Akamai machines split the workload of caching and serving up these files. Each system is built out with a full gigabyte of RAM for top performance, and caches all requests to save bandwidth and further reduce the time required to serve the page. This results not only in peak performance, but also in bandwidth savings on our upstream connections. -Dane and Kelsey

Thu Aug 24 16:46:49 PDT 2000 — Above.net has suffered cuts of multiple cross country fiber circuits in Widner, Georgia. As Above.net is the hosting location for many websites, you may notice some impact for these West coast based online properties. Supernews is hosted at Above.net, and we noticed some impact to the performance of our backup news services from them. Above.net reports now that all three circuits have been restored, and performance is returning to normal. -Dane

Thu Aug 24 14:01:45 PDT 2000 — Update: we are wrapping up the log analysis, and will send out a full list of lost emails to affected customers shortly. Meanwhile, for customers who are more technical, here’s details on the error that caused this.

We have a global procmailrc which controls delivery of email into customer mailboxes. This RC is identical across all servers, but had the ‘HOST’ tag set. We propagated a procmailrc file which contained the hostname for prop.sonic.net over to sub, ultra and buzz, and they began dropping email. This is documented in the procmailrc man page:

PROCMAILRC(5) PROCMAILRC(5)

NAME procmailrc – procmail rcfile

HOST If this is not the hostname of the machine, processing of the current rcfile will immedi- ately cease. If other rcfiles were specified on the command line, processing will continue with the next one. If all rcfiles are exhausted, the program will terminate, but will not generate an error (i.e. to the mailer it will seem that the mail has been deliv- ered).

The host tag is used if multiple machines receive a message, which might be forwarded for example during another procmail delivery, but which some machines are intended to ignore. We have no use for the host tag, and it’s been removed from our configurations. -Dane and Kelsey

Thu Aug 24 13:20:49 PDT 2000 — We have had a system failure that led to lost customer email. Beginning at 9:28:53 AM Thu Aug 24 2000 and ending at 12:15:00 PM, three out of our redundant array of five email servers were misconfigured, and silently discarded customer email. Lost messages were not bounced back to the sender, and cannot be recovered. We are shocked by this event, and have eliminated the configuration which caused this. We will be informing all senders and all recipients of each lost email message with full information including the date, time, sender and recipient of each message so that they can be re-sent.

We sincerely apologize for the error, and the inconvenience that it causes.

In addition to elimination of the configuration setup which caused this, we’re committed to deployment of a testing engine which pro-actively sends and receives email through each of the mail servers individually on an ongoing basis. By testing each and every server independently every few minutes, both locally at Sonic.net and from off site, we can stay better informed about any possible loss or delay which might affect our customers email delivery. This is similar to our existing ‘ckhosts’ and ‘ckdisk’ tools which check for availability of web, mail, ftp, Usenet news services and disk space currently.

These tools page our staff of system administrators with text messages anytime a server becomes unresponsive or unavailable, and this allows us to deliver high availability for your Internet services. We will followup here in the MOTD once this testing engine is deployed.

If you suspect that you may have lost email, please stand by for a notification. -Dane, Scott and Kelsey

Wed Aug 23 17:17:12 PDT 2000 — Just to let everyone know, we’ve increased the maximum email message size from 7MB to 15MB due to continued demand to up the size limit. We also increased the user file size limit on bolt from 24MB to 50MB. -Steve, Kelsey

Sun Aug 20 04:46:16 PDT 2000 — SOLVED: Routing loop on Cable & Wireless. CW has completed their emergency maintenance in San Francisco.Apparently the problem snowballed when they tried to replace an interface card that wasn’t supported by their router’s IOS. (Or something like that — details for their 5-hour outage are vague.) The CW T3 is up now, and I am seeing normal traffic loads. -Scott