Month: October 2000

Tue Oct 10 02:08:26 PDT 2000 — BroadLink has completed the migration of their equipment into a new cabinet here, and all customers have been moved from the cross-connected T3 between Sonic.net and BroadLink over to an ATM T3 via PacBell. Downtime was about one hour and fifteen minutes while BroadLink equipment was moved and customers were reconfigured. This new cabinet space will give BroadLink room to grow, and the new ATM T3 circuit means better growth and management for them as their network continues to grow. They forgot the doughnuts. -Dane, Scott, Shane (BL) and helpers.

Sun Oct 8 16:56:50 PDT 2000 — Name services slow. Some folks (depending on their DNS configuration) have complained about slow DNS services. Investigation reveals that one of our nameserver caches wasn’t working properly. We’ve disabled use of that cache, which cleared the problem. -Scott

Fri Oct 6 02:55:35 PDT 2000 — Strange things afoot this morning. At a little after 1am this morning, a tech support workstation caused a broadcast storm. That workstation is now isolated from our network pending investigation. At this stage, we can’t rule out the possibility of a DoS attack “bounced” off of the machine — but, the workstation is running a new release of a popular operating system, so we can’t rule out a malfunction, either. We’ll post an motd entry when we have more information. -Scott

Thu Oct 5 11:00:27 PDT 2000 — BroadLink will be migrating customers to a new ATM T3 late Monday night beginning at midnight. We expect the coordination, cabling and reconfiguration between Sonic.net and BroadLink to incur a total of about thirty minutes of downtime for BroadLink customers, between around 12:30 and 1am Tuesday morning.

Thu Oct 5 10:52:25 PDT 2000 — An update on the ###-9811 capacity problems. We have contacted our line vendor to have this order expedited for an earlier install date. We will post an update here in the MOTD when we get that new date. As always if you are running into busy signals you can use one of our alternate dialup numbers. Sonic.net has multiple dialup number in every area we serve. To find a list of dialup numbers you can use the Sonic.net pop finder located at www.sonic.net/cgi-bin/pops.pl -Steve

Wed Oct 4 10:56:06 PDT 2000 — We have been experiencing capacity problems on our ###-9811 dialup numbers. We are aware of this and have ordered more capacity for this hunt group. Focal has informed us that the new lines will be online on Oct. 24. Once these new lines are in place the capacity problems will be resolved. -Steve (Note: The problem only manifests itself during “busy hour”, which is Monday night between 8:30 and 9:30 pm. We have taken steps to minimize its impact on customers. -Scott)

Tue Oct 3 08:46:13 PDT 2000 — Regarding the Sunday entry in the MOTD, we should point out that the restoral process tickled a bug in the NetApp, with the result that the individual “.snapshot” directories occasionally pointed to the wrong snapshot. The result: Some people will find other folks web files in their directory. These will go away sometime today, and we apologize for any confusion this may have caused. -Scott

Sun Oct 1 21:10:56 PDT 2000 — On Saturday morning at 7:36am, a Sonic.net customer removed other customer web data files and directories which had insecure permissions set. All data was restored by 1pm Saturday.

Customers had improperly set their permissions so that group ‘user’ had full read/write/execute permissions, meaning that any other user could delete the data. If you are one of the 270 affected users, you will receive an email in the next day or two with some advice on file and directory permissions. Meanwhile, we have removed group write permissions from all restored directories to limit the potential for additional problems in the near term. Please post to news:sonic.help.www or news:sonic.help.cgi if you have any questions about web publishing and permissions.

After an extensive investigation, we were unable to determine which Sonic.net customer deleted the files, but we did narrow it to a group of less than ten potential individuals. It appears likely that this was not a malicious act, and we would appreciate it if the person who did this would step forward with an explanation.

Sonic.net was able to restore all user data from live on-line backups, courtesy of Network Appliance’s “snapshot” feature. All user data is backed up three times daily, and is stored for two days as a “picture in time” of the filesystem. This allows our staff and users to go back to a snapshot from yesterday or the day before and restore edited files to older versions, restore accidentally deleted files, etc. Of course, there is also a nightly backup to our 1.1 Terabyte dual-drive robotic AIT tape library, but this is ‘near line’ storage, and not as convenient for restoration as the NetApp’s snapshot.

The delay in informing users here in the MOTD was due to time involved in investigation and restoration. Followups were made in the newsgroups on this topic as the situation was being resolved. -John, Scott, Nathan, Eli and Dane