Author: admin

Wed Oct 4 10:56:06 PDT 2000 — We have been experiencing capacity problems on our ###-9811 dialup numbers. We are aware of this and have ordered more capacity for this hunt group. Focal has informed us that the new lines will be online on Oct. 24. Once these new lines are in place the capacity problems will be resolved. -Steve (Note: The problem only manifests itself during “busy hour”, which is Monday night between 8:30 and 9:30 pm. We have taken steps to minimize its impact on customers. -Scott)

Tue Oct 3 08:46:13 PDT 2000 — Regarding the Sunday entry in the MOTD, we should point out that the restoral process tickled a bug in the NetApp, with the result that the individual “.snapshot” directories occasionally pointed to the wrong snapshot. The result: Some people will find other folks web files in their directory. These will go away sometime today, and we apologize for any confusion this may have caused. -Scott

Sun Oct 1 21:10:56 PDT 2000 — On Saturday morning at 7:36am, a Sonic.net customer removed other customer web data files and directories which had insecure permissions set. All data was restored by 1pm Saturday.

Customers had improperly set their permissions so that group ‘user’ had full read/write/execute permissions, meaning that any other user could delete the data. If you are one of the 270 affected users, you will receive an email in the next day or two with some advice on file and directory permissions. Meanwhile, we have removed group write permissions from all restored directories to limit the potential for additional problems in the near term. Please post to news:sonic.help.www or news:sonic.help.cgi if you have any questions about web publishing and permissions.

After an extensive investigation, we were unable to determine which Sonic.net customer deleted the files, but we did narrow it to a group of less than ten potential individuals. It appears likely that this was not a malicious act, and we would appreciate it if the person who did this would step forward with an explanation.

Sonic.net was able to restore all user data from live on-line backups, courtesy of Network Appliance’s “snapshot” feature. All user data is backed up three times daily, and is stored for two days as a “picture in time” of the filesystem. This allows our staff and users to go back to a snapshot from yesterday or the day before and restore edited files to older versions, restore accidentally deleted files, etc. Of course, there is also a nightly backup to our 1.1 Terabyte dual-drive robotic AIT tape library, but this is ‘near line’ storage, and not as convenient for restoration as the NetApp’s snapshot.

The delay in informing users here in the MOTD was due to time involved in investigation and restoration. Followups were made in the newsgroups on this topic as the situation was being resolved. -John, Scott, Nathan, Eli and Dane

Tue Sep 26 14:17:11 PDT 2000 — Cidera will be upgrading the service adaptor serving our satellite downlink for Usenet news tonight to add additional streaming media features. Expected downtime is five minutes, and the news server should catch up with new articles shortly after that by using the redundant terrestrial links. -Dane

Mon Sep 25 19:32:15 PDT 2000 — As part of some troubleshooting we are conducting with our Focal lines, we reset a PRI in the Focal hunt group. This reset caused the disconnection of four subscribers, and we’re sorry for that brief interruption. This testing is intended to help us track down a problem with the hunting of our Focal lines. Unfortunately, the trouble only manifests itself during peak periods like Monday night. -Scott and Steve

Fri Sep 22 17:31:28 PDT 2000 — Sunday morning 2:00am we will be shutting down out UUNET circuit for maintenance expected to be completed by 4:00am. Connectivity will still be available through our Cable and Wireless circuit and we will be monitoring it for any problems -Scott and Kelsey

Thu Sep 21 08:52:51 PDT 2000 — We experienced a full mail spool volume this morning and part of last night which may have caused an error message when retrieving e-mail, this error message would look something like “ERR error locking your mailbox” I have cleared up some space on the mail spool and all looks well now. Sorry for any problems this may have caused. No email was lost while this occurred. -Steve

Wed Sep 20 17:58:26 PDT 2000 — On Thursday, September 21st at 2am, we will be rebooting our core router, Mega. Cisco requested we capture diagnostic information during the boot process to troubleshoot a bug we have found with Mega’s operating system. Global connectivity will be unavailable for about 3 minutes. We apologize for any inconvenience. Incidentally, we are well into the planning stages for a redundant router configuration, one in which a core router reboot will not effect global connectivity. -Scott

Tue Sep 12 12:27:14 PDT 2000 — Many BroadLink customers are currently offline, and they are investigating to pin down the trouble. Initially, it looks like it may be a problem with their primary ATM router. Currently, we have no ETR from BroadLink, but they expect to resolve this quickly. Shane is on the way to the Sonic.net data-center to check the equipment now. -Dane, John (BL) and Shane (BL)

Tue Sep 12 02:18:13 PDT 2000 — Tonight’s upgrades. Tonight at midnight, we upgraded our core router (Mega) and the machine serving additional email boxes (“Mailbox”, aka Tsunami).

Mega’s upgrades includes a new operating system, which supports more features in our Cisco 7507. Tsunami’s upgrades include RAM and a faster network card. We also moved mailbox email storage to our Network Appliance, which should improve performance, and provides excellent data assurance. -Kelsey & Scott