Microsoft Windows security issue.

Fri Dec 21 18:44:15 PST 2001 — Microsoft Windows security issue. A major security vulnerability has been announced that affects all MS/Windows XP machines, and which affects Win98/ME machines that have the WinXP ICS (Internet Connection Sharing) package installed.

We’ve noticed a significant slowdown when accessing the Windows Update support site due to load on the remote end, and recommend that customers read up on this latest bug/patch as necessary. News is available at: www.sonic.net/support/security/. -Eli

Broadlink has lost power to the Mt.

Fri Dec 21 13:28:17 PST 2001 — Broadlink has lost power to the Mt. Barham site which serves North Eastern & Eastern Santa Rosa. PG&E scheduled an outage but the site owner failed to inform Broadlink of the down time. This affects approximately 130 customers and the ETR is 3 PM today (PST).

Update: PG&E has restored power to the tower site. BroadLink says sorry about the interruption! -Broadlink and Sonic.net

We are pleased to announce that OpenSRS will…

Thu Dec 20 12:31:23 PST 2001 — We are pleased to announce that OpenSRS will be moving their registry network to a new co-location facility provided by IBM.

To facilitate this move, we will be bringing our domain registration and management tools off-line on Saturday, December 22nd from 2:00am PST to 6:00am PST.

-Chuckc

The OpenSRS system experienced a brief but…

Wed Dec 19 11:58:34 PST 2001 — The OpenSRS system experienced a brief but total system outage today from 09:30-09:40 PST. All systems are now responding normally. OpenSRS has isolated the cause, and made appropriate changes to ensure that this does not happen again. We apologize for any inconvenience this may have caused you. -Chuckc

Bandwidth Quota Updates: We now have full…

Wed Dec 19 14:21:31 PST 2001 — Bandwidth Quota Updates: We now have full accounting for SSL web service traffic included in hedgehog and Webalizer. All SSL related quotas are being properly accounted for at this time. Reports and stats will start appearing tomorrow, Dec 20th. -Kelsey and Nathan

UUNet T3 down, backup link functioning.

Wed Dec 12 06:06:55 PST 2001 — UUNet T3 down, backup link functioning. Our 45Mbps T3 to UUNet is currently down, and UUNet is working with PacBell now to isolate the trouble and repair it. Meanwhile, our 15Mbps link to Cable & Wireless is functioning well, thought it is pretty full. Internet performance this AM may be a bit sluggish until PacBell repairs the link to UUNet.

Update, Wed Dec 12 07:03:23, Pacific Bell has isolated the trouble as a loop-back in the Santa Rosa central office, and has dispatched a technician to that location to check the circuit. No ETA is available from Pacific Bell for the technician’s arrival. Currently, Internet connectivity is available, but packet loss is running 10%-20%, and performance is sluggish.

Update, Wed Dec 12 08:05:29, UUNet reports that they’ve escalated to tier three support at Pacific Bell, and that Pacific Bell’s projected ETA at the Santa Rosa central office for the technician was 7:30AM PST. As of 7:50AM, they had not heard from the repair technician with any new information. Currently, Internet connectivity is available. Packet loss is running around 13% now, and latency is about half a second longer than it should be. -Dane

Update: For a few of network statistics the UUNet outage, see:

eye.sonic.net/public_cricket/grapher.cgi?target=%2Frouter-interfaces%2Fall%20uplinks%20%28sum%29;ranges=d%3Aw;view=Octets

Note that we’ve got 15Mbps to Cable and Wireless, which is working fine, but our primary 45Mbps link to UUNet is down. As you can see from the graph, we’ve got less capacity than we need at this point, so we’ve got functioning Internet connectivity, but it’s performance is likely to become more degraded as the morning goes on.

For a comparison of utilization of the links, see the following graph. Note the green and dark blue is UUNet, which drops to zero at around 5:30AM, and the red and light blue is Cable and Wireless, which immediately jumps to near capacity.

eye.sonic.net/public_cricket/grapher.cgi?target=%2Frouter-interfaces%2Fall%20uplinks%20%28individual%29;ranges=d%3Aw;view=Octets

Sonic.net’s stats pages provide extensive network health information, and are available at www.sonic.net/stats/. -Dane

Update, Dec 12 09:17:00: We are on the phone with a Pac Bell tester and UUNet support. They suspect the problem is between the Santa Rosa CO and Sonic.net. Further updates will follow as information becomes available. -Matt, Dane and Steve

Update, Dec 12 10:05:16: There is now a Pac Bell tech working on the circuit in the Santa Rosa CO. We are on the phone with him and UUNet narrowing this down. -Matt

Update, Wed Dec 12 11:05:01: There are apparently two problems with the circuit. Pac Bell is working on the DSX3 in the Santa Rosa CO and dispatching a technician to Sonic.net. -Matt, Dane and Kelsey

Update, Wed Dec 12 12:11:30: Pac Bell isolated to problem to a bad DSX card and has replaced the card. This seems to have resolved the problem and traffic has shifted back to the UUNet circuit. Packet loss is gone and latency has returned to normal. We will be keeping a close eye on the T3 but expect that the problem is resolved. -Matt, Dane, Scott, Kelsey, Steve, Russ and Eli

Updates to Hedgehog and Quotas: We have made…

Mon Dec 10 17:26:11 PST 2001 — Updates to Hedgehog and Quotas: We have made a number of modifications, improvements and bug fixes to our quota tracking and billing software. 1) We are now exempting local web and ftp traffic from being counted against users’ quotas. This is currently reflected with the ‘Web_local’ service. 2) We have also embedded full web and ftp traffic analysis into Hedgehog using Webalizer. The Webalizer reports can be retrieved by clicking on the ‘Domain’ link while looking at the bandwidth and hit summaries.

3) We also enabled incremental billing for bandwidth in 100MB chunks instead of full 1GB chunks. ($15.00/GB $1.50/100MB)

4) We have added the most requested features since the release of Hedgehog as well. We now offer bandwidth quota protection which will, if your bandwidth quota is exceeded, redirect website visitors to a ‘User Over Quota’ page and lock out access to your FTP space. Users can enable this feature on their accounts with Hedgehog in the member tools.

We are also going to change the layout for the way that we store our raw weblogs on our servers as well. Currently, weblogs are stored in /var/log/httpd/domain.name. If a user wishes to review their weblogs for their personal web space they have to sort through all of our users ~3 million hits to find their own webpages. We will be moving the format to /var/log/httpd/USERNAME/domain.name. Splitting out the weblogs into directories by username as well as domain should make it easier for our users to access their weblogs, as well as providing extra security by preventing users from being able to see every other users’ weblogs.

If you have any other suggestions or feedback please post them to news://news.sonic.net/sonic.net -Kelsey and Nathan