Month: January 1998

Thu Jan 29 12:51:42 PST 1998 — Access has been restored in the South Bay, and all’s well. The OC48 circuit to MFS is back online. -Dane

Thu Jan 29 12:27:50 PST 1998 — There was a small fire in our South Bay facility, and access is currently offline. All bay area numbers are unavailable. At approximately 11:00 this morning, a monitoring box installed by MFS for their OC48 equipment ignited. MFS is on site, and is working with Above.net, Brooks and TCG to restore full access. -Dane

Thu Jan 29 17:55:39 PST 1998 — We’re still having problems with Pacific Bell’s provisioning and USR’s implimentation of the DMS ISDN switch protocol, and we haven’t made much progress with it today. Pacific Bell has promised that they will make more headway tommorow, and we’re hoping to reprovision at least some of our lines on Saturday morning at 2am. Meanwhile, we have completed deployment of quite a bit of new equipment, and we do have available capacity. If you experience any problems getting connected (busy signals, reorder tones, etc), please try calling 522-1003 if it’s local to you. If you have any feedback or comments, please post in the sonic.help newsgroup. -Dane

Wed Jan 28 18:30:47 PST 1998 — We’ve isolated the problem with false busy signals as a bug in the implimentation of the DMS 100 custom switch protocol in USR’s MP16I equipment. We’ve asked PacBell to reprovision all of the lines which are affected (104 BRI lines, 208 ports) to the National ISDN 2 switch protocol, and they are coordinating and scheduling this now. If you are hearing a fast busy, also known as a reorder tone, you can avoid this problem by dialing ‘*82,’ before the access number you are calling. This problem affects customers in almost all of our v.32 and X2 points of presense, including customers calling 522-1001. If you suspect that you may be affected by this, use a regular telephone on the phone line which you use for your modem and dial 522-1002, the line which serves X2, plus overflow for v.34 callers. If you hear the fast busy, add ‘*82,’ to your dial string. We’re hoping that PacBell can coordinate the reprovisioning of these lines for tommorow morning (Thursday AM) between 2AM and 6AM. During this time, 522-1002 will be unavailable, as we’ll be reprogramming all of the modems with the new switch protocol and new SPID numbers. We will probably also take advantage of this opportunity to flash upgrade the X2 gear to the latest firmware, as we’re running 2.1.5 and 2.2.2 was recently released.

Tue Jan 27 17:32:39 PST 1998 — We’ve got partial deployment of our new ISDN PRI access server, nas21, a USR/3Com Total Control Enterprise Network Hub. This unit now serves overflow for analog and X2 callers in Sonoma County, plus dual-B ISDN callers. We’re continuing to have problems with real and false busy signals. The real busy signals should be gone with the partial deployment (46 ports) of the new nas21, and we’ll be finishing deployment (92 ports total) of that hardware tommorow. The false busy signals are due to problems with caller ID blocking. If you are experiencing busy signals, add ‘*82,’ before the access number you are calling. We’re working with Pacific Bell and USR to address this issue. -Dane

Sun Jan 25 10:49:24 PST 1998 — Our dedicated T1 circuit to MCI was offline today from about 1AM until 10:30AM because of what appear to be Pacific Bell line problems or MCI backhaul problems. Because of our investment in fully redundant intelligent routing, no customer services were affected. Our currently connectivity lineup includes an optical T3 circuit to UUNet in San Jose, and metallic T1 circuits to MCI in Sacramento and to Sprint in Stockton. All of these are handled by our huge Cisco 7000 series router. The Cisco decides which circuit is the shortest path to any given Internet destination, and all of these decisions are dynamic. Of course, if a circuit is offline, it’s not the shortest path, so all traffic is automaticly re-routed. Additional routes include T3 peering with DNAI, and T1 peering with NapaNet and a2i. -Dane

Sun Jan 25 10:43:48 PST 1998 — Our Network Appliance upgrade went well, with total downtime of 21 minutes between 4:05AM and 4:26AM. The Network Appliance uses it’s main memory to cache disk array read requests, and with the 64 megs in the default configuration, it could only cache about 2 minutes worth of data. It’s now caching 12-14 minutes worth of read data, which means that it will get many more cache hits and thus less time waiting for disks. In addition, we upgraded the write cache from 2 megs to 8 megs of non volatile RAM (NVRAM). The battery backed NVRAM is used to buffer writes so that client hosts receive confirmation of a write immediately, without waiting for disks. Previously, the two megs was getting filled every 2-3 seconds, forcing a write to disk. Now, we’re just writing at the 10 second ‘checkpoint’, which is great. The Network Appliance is currently averaging about 300-400 operations per second, mostly from the main web server’s 1+ million hits per day. Thanks to Scott for handling the late upgrade. -Dane

Sat Jan 24 01:24:49 PST 1998 — We’ve rescheduled upgrades for our Network Appliance NFS filter for Sunday morning at 4am. As this filer handles storage for many services, local shell, mail and web hosting will be offline for about 10 minutes for this upgrade. We will be upgrading the RAM and NVRAM in the filer to it’s maximum capacity in order to enhance performance. Have a nice weekend, and enjoy the Superbowl! -Dane

Sat Jan 24 01:22:19 PST 1998 — Our news server is having some troubles this evening. It crashed at about 9pm, and as of now, it’s still checking it’s massive filesystems. During this time, users can read news, and make posts, but the posts won’t be injected into the system until the check is complete. Don’t be surprised if posts take a while to show up on the system this evening. -Dane

Tue Jan 20 19:41:32 PST 1998 — We’ve just added more drive space to our Network Appliance NFS filer. The filer now has eight four gig disks in service, including the parity disk and a hot spare. This upgrade was done during heavy utilization, and because of the way the NetApp is designed, no services were interrupted. The disk is plugged in, the filer spends 15 minutes preparing it, and then all of the mounted hosts have more space! Zero downtime. Here’s before and after df output: Filesystem kbytes used avail capacity Mounted on / 17237284 16438352 798932 95% / And 20 minutes later, after the disk is installed: / 20684744 16310496 4374248 79% /

The filer has room for four more four gig disks in it’s current storage shelves, and we can add two more shelves to the unit in the future for a total raw storage capacity of about 96 gigs. -Dane