Wed Dec 1 12:01:54 PST 1999 — During network reconfiguration that should have been non-intrusive, our core gateway router, mega.sonic.net crashed and reloaded. We were changing the pathing to re-balance load on our circuit to Cable and Wireless as we work on ongoing latency troubles on our UUNet T3, and this isn’t an operation that should cause any problems in the router.

We’ve opened a support ticket with Cisco, and have asked their tech group to come up with some sort of reason for what happened. Downtime was four minutes and 42 seconds, during which Internet connectivity was now available. We’re very sorry for the interruption, and we’ll work with Cisco to come up with an explanation and to try to come up with some way to achieve more redundancy than we already have in our core structure. -Dane, Scott and Co.

Wed Dec 1 19:52:11 PST 1999 — Urchin, the new web log analysis tool, is up and running. You may take a look at your multihome’s urchin report by going to www.sonic.net/stats and filling out the form. You will be prompted for the username and password associated with the multihome. The raw web logs have also been updated to include November 30th. Both Urchin and the raw logs are updated once a day around 2:00am. -Kelsey

Tue Nov 30 12:00:51 PST 1999 — In the ongoing troubleshooting of the latency on our primary T3, PacBell and UUNet have found a buffer setting on the UUNet end of the circuit which needed to be increased. As we’ve been experiencing less than ideal performance for some time, we elected to make this quick change during the day time. It’s complete, and during the shift, we’ve seen slow performance as the network converges. While convergence is still in process, I am seeing quite good performance, so this is encouraging.

While the network is converging, performance is less than ideal. Thank you for your patience. -Dane

Tue Nov 30 08:40:39 PST 1999 — We will be performing maintenance on a number of systems on Friday night, beginning at midnight. A number of machines are being upgraded, but most user services will be unaffected. However, serving of websites will be interrupted briefly as we upgrade these machines, and migrate them to a new layer 4 switch. This new switch allows us to balance load between many web servers, and means that we can down a server for maintenance (or have one crash due to hardware or software failure) without any perceived impact from the browsing end-user. This is a great enhancement to our hosting offering, and provides our customers with a really nice degree of redundancy. The L4 switch is Alteon’s AceDirector-3, a fast hardware based application layer switch. This ‘behind the scenes’ investment in our infrastructure will pay off for our customers as we continue to scale our offerings without excessive downtime or performance bottlenecks.

Tue Nov 30 08:38:11 PST 1999 — The mail server that handles inbound email for ‘sonic.net’ had some troubles overnight, and stopped accepting messages at 21:57 last night. I’m still trying to isolate the cause of the stoppage; all of the services were running fine, and there was no obvious reason for it to stall. It’s receiving piles of overnight email now, we’re very sorry about the delayed delivery of some emails! -Dane and Dom

Mon Nov 29 17:51:23 PST 1999 — In our continuing effort to provide the best Internet service possible and to make way for the new load balanced web servers we are making some changes to the way our web servers handle log files. Starting Dec first we are replacing the existing web log analysis tool with Urchin. Urchin is a great log analysis tool which runs daily building html reports and sends you weekly and monthly status reports via email. Customers who have been accessing their raw log files will notice that the raw logs are no longer being generated in real time. They will be updated and rotated at the end of every day (you will see Tuesday’s logs Wednesday morning.) We will provide a tool for customers that enjoy watching their log files with a ‘tail -f’ in the next few days. -Kelsey

Wed Nov 24 18:23:35 PST 1999 — I will be upgrading memory in thunder.sonic.net tonight to half a gigabyte. This will further improve performance for many multihomed websites. We are working now on deployment of a number of improvements for multihomed hosted sites including a really nice new stats package and a large new Alteon AceDirector 3 layer four switch. The L4 switch allows us to balance load between multiple web servers, which increases reliability and also gives us the ability to down servers for upgrades and maintenance transparently.

Multihomed customers already have enhanced quotas – an extra 50 megs of storage for each site, and an extra 200 megs per day of outbound served data transfers. We’ll continue to work to add value to this great service offering. -Dane, Kelsey and Scott

Wed Nov 24 18:19:17 PST 1999 — We are again seeing latency problems on our primary T3 circuit, and we’re working with PacBell and UUNet to isolate the trouble. PacBell will be testing the circuit tonight at 2am to see if they can find any problems, so we expect slower than usual connectivity as we run on backup links at that time. We’ll keep you posted as we continue to work on this. -Dane

Tue Nov 23 08:31:04 PST 1999 — At 3am last night, PacBell replaced the interface card which serves our UUNet T3 connection. This appears to have eliminated the latency troubles we’ve been seeing in the past few weeks. I’m currently seeing 17ms average latency over this circuit. We were previously seeing performance as bad as 120-160ms average. -Dane

Tue Nov 23 08:28:00 PST 1999 — Note that when you connect to the new news server the first time, you will probably be offered a few thousand new groups. Using most shell based newsreaders, you’ll be asked if you want to join each one, with a yY/nN option. Answering ‘N’ will skip all of the rest of them, answering ‘Y’ will join all of them. Lowercase n/y will allow you to work through each of them, but this can get really tiresome as there are so many. Most users will want to just hit ‘N’.

The new server has some article numbering issues as a result of the transition, as the numbers are close, but not exactly what was on the old server. Because of this, some groups may have different numbers of articles than the group listing indicates. For more discussions regarding the new server, see ‘news:sonic.net’. -Dane