Thu Nov 8 11:17:40 PST 2001 — We goofed up on the rollout of our new bandwidth monitoring and billing tools. I am sorry about the confusion and conflicting messages about this.

As we published here in September, we have made changes to both our bandwidth and disk quotas, and those changes were to be implemented in October. The disk quota change is simple – we went from 50 megs per service up to 80, and set pricing at $10 per month for each additional 80 megs. “Per service” means for each basic account or multihomed web hosting, so if for example you have an account with a multihome, you now have 160 megs of storage space.

The changes to the bandwidth quota are a bit more complex. In summary, we reduced both the size of the quota and the cost for going over it. In the past, the quota was 200 megs per day, and usage over the quota was $5 per 100 megs, or $50 per gigabyte. Under the new quota, you get 1000 megs (one gigabyte) per month, and the charge for going over the quota is $15 per gigabyte.

This reduction in the fee for overage brings us to a much more fair cost for sites that exceed the quota. In the past, it was far cheaper just to buy another account and split your website up than it was to simply manage it under one account. This never made a whole lot of sense, and caused extra work for for customers seeking to avoid the fees. Sites which did go over the daily quota could quickly accrue charges which were very, very high, and which were not a good reflection of the costs to Sonic.net.

In other words, $50 per gigabyte is too much to charge, the $15 price is a better reflection of our costs. Additionally, providing 200 megs per day to a site that used 150-200 per day both left the customer awful close to treading over that line and having a large charge added to their account, and left Sonic.net moving almost six gigabytes of bandwidth, which had a lot of costs for us.

Now, to the warning system. The intent is that if your site is projected to exceed the quota, you would receive a warning about potential costs. So, assuming that there are 30 days in an month for this example, if during the analysis on the third night of the month, you’d used more than 10% of the total quota, you’d get a warning telling you that we project that you may exceed the quota. This would provide ample time to make changes to the site to reduce it’s size, or find a way to produce some revenue to pay for the bandwidth.

The problem was that the warnings didn’t get sent on a consistent basis last month. Of the 90 or so customers who are over quota, only about ten of them got anything from us warning them about the usage.

Obviously, it’s simply not fair to make changes to the quota pricing and billing methods, then to charge a customer without giving them some warning prior to their incurring the charges. This was not our intention, and I’m very sorry that we posted these charges!

Bandwidth charges for customers who were not warned last month will be removed from their accounts today, and we’ll assure that warnings are sent out properly this month. Sorry about the confusion, and the contradicting messages here in the MOTD regarding this.

On an ongoing basis, we’d like to encourage customers to publish busy websites which do use bandwidth, but we need to work out ways to keep the usage at levels where the costs involved match up with revenue to cover the costs. Discussion is ongoing in news://news.sonic.net/sonic.general currently on this topic.

-Dane, Scott and Kelsey

Wed Nov 7 12:07:53 PST 2001 — .BIZ is live!! OpenSRS has completed their initial testing. They report a stable network and have opened for .biz-ness.

Customer interested in obtaining a .biz domain name can use the Sonic domain tool to process the registration and add domain services.

The domain tool is located at: sonic.sonic.net/members/domain_tool/

-Chuckc

Wed Nov 7 11:09:23 PST 2001 — Bandwidth quota charges. At 5pm yesterday evening, we ran the new bandwidth quota tool. Unfortunately, it appears that some folks had not received the previous warning, resulting in an unexpected charge for the extra bandwidth that they had used. Sonic.net recognizes the inconvenience of receiving a charge without warning; however, we also recognize that most of those billed should have had some idea that they would be over quota. However, Sonic.net wishes to retain the good will of its customer base — toward that end, we are applying a 50% good-will credit across-the-board for current bandwidth quota charges. Disk quota charges are unaffected, as folks over-quota have been warned about disk usage for months.

Please remember that the bandwidth quotas have always been a policy at Sonic.net, but we hadn’t implemented a procedure to collect for over-quota usage. This means that some users have been enjoying quite a bit of free bandwidth with their web hosting. There is bound to be contention as we implement the procedure for over-quota charges; hopefully the good-will credit will smooth the transition to the new system. -Scott, Jen, Kelsey

Tue Nov 6 19:55:33 PST 2001 — The new .biz domain names are going live tonight at 9pm PST. OpenSRS, the registrar Sonic.net partners with, experienced problems during the opening of the .info TLD and has notified us that they will not be taking registrations for the .biz TLD this evening because they cannot reliably handle the unusual load that new domain offerings put on their current system. We are in contact with OpenSRS and will keep you updated with information as it becomes available. We apologize for the delay! -Matt, Chuck and the Sonic.net Staff

Fri Nov 2 11:42:22 PST 2001 — Our MegaPOP number for Phoenix, AZ is returning ‘all circuits busy’ messages, and MegaPOP is working to resolve that issue. MegaPOP has not released an ETR, but repairs are expected later today. We apologize for any inconvenience. – Eli

Update: MegaPOP was able to resolve this issue and doesn’t expect further interruptions. -Matt

Fri Nov 2 16:20:38 PST 2001 — BroadLink has resolved some intermittent latency issues which were affecting customers in some areas. Firmware upgrades were made to some customers radios which were flooding the network and causing the problem. -Dane and BL

Thu Nov 1 02:11:19 PST 2001 — We’ve completed the firmware upgrade on the Extreme Networks. The upgrade actually required two reboots which took about three minutes each.

After the upgrade, our 200Mbps FD Etherchannel connections to the Alteon load balancing switch which serves our mail and web servers seems to have failed. We’ve backed up to a set of 100Mbps links, and the servers are available.

We’re going to take this opportunity to upgrade the internal backbone link between the primary Alteon and the core switch to gigabit Ethernet, which will address our issues with Etherchannel between the Black Diamond and the Alteon. Once we’ve got that working, we’ll be bringing online “hot”, a new Extreme Networks Summit switch. Once connected, we’ll move the servers from ape (the Black Diamond) over to hot. The move to Gig-E and deployment of hot should include only momentary interruption of web hosting services.

This architectural change allows us to resolve some issues in the Extreme and Alteon environment where the Alteon was acting as a load balancing switch between two vlans in ape. This type of back and forth of traffic was causing some confusion in the switches, and caused some headaches.

-Kelsey and Dane

Wed Oct 31 11:14:52 PST 2001 — We will be upgrading firmware on our primary core switch at 1AM Thursday morning. Downtime is expected to be two to three minutes while the switch reboots after the upgrade. The bug fixes in this release should resolve some of the instability that we’ve seen in the product which have caused downtime in the past. We’ll also be bringing our redundant management switch module back online for full “brain” redundancy in this switch.

Our huge Extreme Networks Black Diamond 6808 switch is the primary core switch here at Sonic.net, and much of our network is served by it. A redundant Extreme Networks switch serves one of the two primary Cisco 7206VXR routers, and parts of the network will be reachable during the upgrade, including T3 and some T1 customers. -Dane and Kelsey

Wed Oct 31 17:07:18 PST 2001 — Our October domain registration special expires tonight, so if you’re interested in securing a domain name under the special pricing, see the tools at:

sonic.sonic.net/members/domain_tool/

-Chuck and Dane

Tue Oct 30 09:39:14 PST 2001 — shell.sonic.net has been moved to our new shell server, newbolt.sonic.net. bolt.sonic.net points toward this system now as well, but please always use shell.sonic.net as the primary name for shell service.

The old shell server was halted last night at 12:30AM after we discovered that the ssh host key had changed, implying that the system may have been broken into. It’s been quarantined for forensic analysis, after which we’ll have a better idea of what occurred.

The new shell server is a massive system with dual Pentium III 1 Ghz CPUs, almost a gigabyte of RAM and fast new drives. If you have any software requests or discussions, please post to: news://news.sonic.net/sonic.net

Currently, ssh access to the shell server is available, but telnet and rlogin are currently locked. We expect to turn on rlogin access from our dialup terminal servers in the next hour or so. Telnet access will be turned on shortly, but we’re likely to eliminate it in the next few days in favor of ssh, at least from outside our local network. If you don’t currently have an ssh client such as putty or terraterm, now would be a good time to seek one out and install it.

Update: rlogin access (including shell dialup) is still in the works. We’re having some trouble integrating it into our authentication system. Shell dialup should be available in an hour or two.

Update: rlogin access from our dialup servers (dialup directly to shell) is available. -Dane, Kelsey, Eli, Scott and Steve