Sun Oct 1 21:10:56 PDT 2000 — On Saturday morning at 7:36am, a Sonic.net customer removed other customer web data files and directories which had insecure permissions set. All data was restored by 1pm Saturday.
Customers had improperly set their permissions so that group ‘user’ had full read/write/execute permissions, meaning that any other user could delete the data. If you are one of the 270 affected users, you will receive an email in the next day or two with some advice on file and directory permissions. Meanwhile, we have removed group write permissions from all restored directories to limit the potential for additional problems in the near term. Please post to news:sonic.help.www or news:sonic.help.cgi if you have any questions about web publishing and permissions.
After an extensive investigation, we were unable to determine which Sonic.net customer deleted the files, but we did narrow it to a group of less than ten potential individuals. It appears likely that this was not a malicious act, and we would appreciate it if the person who did this would step forward with an explanation.
Sonic.net was able to restore all user data from live on-line backups, courtesy of Network Appliance’s “snapshot” feature. All user data is backed up three times daily, and is stored for two days as a “picture in time” of the filesystem. This allows our staff and users to go back to a snapshot from yesterday or the day before and restore edited files to older versions, restore accidentally deleted files, etc. Of course, there is also a nightly backup to our 1.1 Terabyte dual-drive robotic AIT tape library, but this is ‘near line’ storage, and not as convenient for restoration as the NetApp’s snapshot.
The delay in informing users here in the MOTD was due to time involved in investigation and restoration. Followups were made in the newsgroups on this topic as the situation was being resolved. -John, Scott, Nathan, Eli and Dane