Month: December 2001

Wed Dec 19 14:21:31 PST 2001 — Bandwidth Quota Updates: We now have full accounting for SSL web service traffic included in hedgehog and Webalizer. All SSL related quotas are being properly accounted for at this time. Reports and stats will start appearing tomorrow, Dec 20th. -Kelsey and Nathan

Wed Dec 12 06:06:55 PST 2001 — UUNet T3 down, backup link functioning. Our 45Mbps T3 to UUNet is currently down, and UUNet is working with PacBell now to isolate the trouble and repair it. Meanwhile, our 15Mbps link to Cable & Wireless is functioning well, thought it is pretty full. Internet performance this AM may be a bit sluggish until PacBell repairs the link to UUNet.

Update, Wed Dec 12 07:03:23, Pacific Bell has isolated the trouble as a loop-back in the Santa Rosa central office, and has dispatched a technician to that location to check the circuit. No ETA is available from Pacific Bell for the technician’s arrival. Currently, Internet connectivity is available, but packet loss is running 10%-20%, and performance is sluggish.

Update, Wed Dec 12 08:05:29, UUNet reports that they’ve escalated to tier three support at Pacific Bell, and that Pacific Bell’s projected ETA at the Santa Rosa central office for the technician was 7:30AM PST. As of 7:50AM, they had not heard from the repair technician with any new information. Currently, Internet connectivity is available. Packet loss is running around 13% now, and latency is about half a second longer than it should be. -Dane

Update: For a few of network statistics the UUNet outage, see:

eye.sonic.net/public_cricket/grapher.cgi?target=%2Frouter-interfaces%2Fall%20uplinks%20%28sum%29;ranges=d%3Aw;view=Octets

Note that we’ve got 15Mbps to Cable and Wireless, which is working fine, but our primary 45Mbps link to UUNet is down. As you can see from the graph, we’ve got less capacity than we need at this point, so we’ve got functioning Internet connectivity, but it’s performance is likely to become more degraded as the morning goes on.

For a comparison of utilization of the links, see the following graph. Note the green and dark blue is UUNet, which drops to zero at around 5:30AM, and the red and light blue is Cable and Wireless, which immediately jumps to near capacity.

eye.sonic.net/public_cricket/grapher.cgi?target=%2Frouter-interfaces%2Fall%20uplinks%20%28individual%29;ranges=d%3Aw;view=Octets

Sonic.net’s stats pages provide extensive network health information, and are available at www.sonic.net/stats/. -Dane

Update, Dec 12 09:17:00: We are on the phone with a Pac Bell tester and UUNet support. They suspect the problem is between the Santa Rosa CO and Sonic.net. Further updates will follow as information becomes available. -Matt, Dane and Steve

Update, Dec 12 10:05:16: There is now a Pac Bell tech working on the circuit in the Santa Rosa CO. We are on the phone with him and UUNet narrowing this down. -Matt

Update, Wed Dec 12 11:05:01: There are apparently two problems with the circuit. Pac Bell is working on the DSX3 in the Santa Rosa CO and dispatching a technician to Sonic.net. -Matt, Dane and Kelsey

Update, Wed Dec 12 12:11:30: Pac Bell isolated to problem to a bad DSX card and has replaced the card. This seems to have resolved the problem and traffic has shifted back to the UUNet circuit. Packet loss is gone and latency has returned to normal. We will be keeping a close eye on the T3 but expect that the problem is resolved. -Matt, Dane, Scott, Kelsey, Steve, Russ and Eli

Mon Dec 10 17:26:11 PST 2001 — Updates to Hedgehog and Quotas: We have made a number of modifications, improvements and bug fixes to our quota tracking and billing software. 1) We are now exempting local web and ftp traffic from being counted against users’ quotas. This is currently reflected with the ‘Web_local’ service. 2) We have also embedded full web and ftp traffic analysis into Hedgehog using Webalizer. The Webalizer reports can be retrieved by clicking on the ‘Domain’ link while looking at the bandwidth and hit summaries.

3) We also enabled incremental billing for bandwidth in 100MB chunks instead of full 1GB chunks. ($15.00/GB $1.50/100MB)

4) We have added the most requested features since the release of Hedgehog as well. We now offer bandwidth quota protection which will, if your bandwidth quota is exceeded, redirect website visitors to a ‘User Over Quota’ page and lock out access to your FTP space. Users can enable this feature on their accounts with Hedgehog in the member tools.

We are also going to change the layout for the way that we store our raw weblogs on our servers as well. Currently, weblogs are stored in /var/log/httpd/domain.name. If a user wishes to review their weblogs for their personal web space they have to sort through all of our users ~3 million hits to find their own webpages. We will be moving the format to /var/log/httpd/USERNAME/domain.name. Splitting out the weblogs into directories by username as well as domain should make it easier for our users to access their weblogs, as well as providing extra security by preventing users from being able to see every other users’ weblogs.

If you have any other suggestions or feedback please post them to news://news.sonic.net/sonic.net -Kelsey and Nathan

Mon Dec 10 16:57:04 PST 2001 — New email filtering tools updated: We resolved the bug that was preventing some users from being able to access the new email filtering tool. -Kelsey

Mon Dec 10 15:02:27 PST 2001 — We’ve expanded the focal hunt group by 15%. This will eliminate occasional busy signals on the ###-9811 dial pool. We have also replaced a PRI in the 522-1003 dialup pool, which returned an additional 23 dialup lines to service. -Steve

Mon Dec 10 13:53:38 PST 2001 — New email filtering tools. We are in the process of evaluating SpamAssassin for integration into our suite of email filters. For the past few weeks our early testers have reported good success with SpamAssassin and we feel that it is ready for more widespread testing by our users. To make it easier for our users to evaluate this too we’ve created a new membertool, ‘The SpamCan,’ which can be used to turn SpamAssassin on or off. The SpamCan can also be used to toggle some of our other email filters as well. We intended to have all of our email filters SpamCan enabled shortly.

SpamAssassin tags email which it believes to be spam by adding headers and by modifying the subject (which can be turned off) and then includes a detailed explanation of why the mail was tagged in the message body. SpamAssassin is fully user configurable; users are able to add whitelist and blacklists, as well as modifying scores and thresholds. In our own testing, SpamAssassin has demonstrated itself to be over %95 effective in correctly tagging spam. Users are then able to use procmail or mail filters built into their email clients to filter the tagged spam to a junk mail folder. We do not recommend deleting tagged messages.

Before enabling SpamAssassin on your account or if you have questions about it or any of our email filters please review the local newsgroup news://news.sonic.net/sonic.antispam and post your questions there. If you are currently using SpamAssassin inside of your own procmail.rc we recommend that you use our tools to turn it on or off instead. -Kelsey

Sun Dec 9 13:07:25 PST 2001 — Circuit failure on the 522-1003 hunt group caused ‘All circuits are Busy’ messages and fast busy signals this morning. After rebooting some of our gear the problem went away, we do have PacBell investigating on their end to see what caused the failure. -Steve

Sun Dec 2 01:11:57 PST 2001 — Night Operations Complete. We cleaned the switch’s configuration and rebooted without any trouble. Hopefully this will clear up some of the stability issues. -Kelsey and Matt

Sat Dec 1 22:08:33 PST 2001 — We will be rebooting ape, our core Extreme Networks switch tonight. It’s configuration has become corrupted, most likely over the course of software upgrades done in the past. EN believes that the corruption in the configuration may be the cause of some of it’s stability problems. We’ll bring the switch down shortly after midnight and total down time shouldn’t be more than 15 minutes. This procedure will take longer than a simple reboot because the switch must be brought up ‘dumb’ with a default configuration before the cleaned configuration can be applied to it. -Kelsey, Nathan and Matt

Sat Dec 1 17:44:31 PST 2001 — In a move that peripherally affects Sonic.net customers, Excite@Home cut off Internet access for AT&T Broadband cable modem customers on Saturday morning at about 2:15AM. With about 850,000 end-users affected, this is a major impact for broadband end-users. Included in these are cable modem customers in Petaluma and Windsor.

Here at Sonic.net, we’ve noticed a large decline in the amount of outgoing traffic, as @Home was one of the largest single consumers of content from sites hosted here. If you have your own website here, you may notice less traffic.

We have also had some user calls regarding websites which were hosted by @Home customers on their cable modem connections (side note – this isn’t allowed by the AT&T terms of service) which are now offline.

To check if a site that you cannot reach is on the @Home network, use our Ping/Traceroute/MTR tools at www.sonic.net/stats/. If the address is behind a router at “home.com”, and is unreachable, it’s likely that it is an AT&T cable modem customer who has been cut off.

AT&T has told customers that they can expect to be without service for “a few weeks” (CNet). Sonic.net has received DSL orders from some AT&T customers to replace their cable modem service, and we’re glad to be bringing these customers back online with us directly! -Dane