Year: 2002

Sat Feb 23 00:25:54 PST 2002 — We will be performing maintenance on the gear that serves 522-1003 tonight from about midnight through 2:00am This may cause some people to get disconnected and also may result in ‘all circuits are busy’ messages. We are upgrading the firmware on our gear in an attempt to resolve the ongoing issues with 522-1003 -Steve

Wed Feb 20 17:02:29 PST 2002 — After a lot of head-scratching, we may have a line on the 1003 trouble. Our vendor thinks that the code release we are running on the USR gear may be using too much memory. We have taken immediate steps to reduce the memory footprint on the USR gear, and will be flashing the gear with new code tonight at midnight. -Scott and Steve

Mon Feb 18 04:43:40 PST 2002 — We’ve had troubles with our 522-1003 dialup again this evening. As always, we’ve got backup dialup numbers in virtually all of our service areas which can be found using the “Find Dialups” link on the home page. We’d suggest printing this list in case of emergencies such as this one.

The past two weeks have been very bad with dialup troubles, primarily in our USR equipment serving the 1003 dialup. We’re trying to isolate the source of the problems! -Dane and Eli

Update Wed Feb 20 08:34:41 2002 — 522-1003 was returning all circuits are busy this morning. After a reboot the problem went away. We are working with our hardware vendors and PacBell to pin point the source of this problem. -Steve

Update Wed Feb 20 13:34:24 2002 — Another failure on 1003. We are closer to finding a solution. We have a PacBell tech monitoring our T3 and collecting data when the failures happen. -Steve

Update Wed Feb 20 15:19:48 2002 — The 8th PRI out of 40 or so in 1003 failed, and we found that an unplug/replug of the PRI cable fixed the issue. PacBell seems to be pointing toward the USR equipment as the source of the trouble, but the failures are not isolated to a single bit of equipment here and they’ve all worked well for years previously. We’re continuing to escalate at USR for diagnostic assistance. More to following in the MOTD. -Dane

Update Fri Feb 22 07:50:14 PST 2002 — 2 failures of 1003 this morning caused the all circuits are busy error, we had to reboot one of our USR boxes to clear the problem both time. The changes we made on Wednesday did not appear to help with this problem. We are still working with USR engineers to resolve this problem quickly. -Steve

Thu Feb 14 15:24:11 PST 2002 — Un-scheduled domain registration outage. We have disabled the domain tool due to an un-scheduled outage of the .com/.net./org registry. At this time, we have isolated the source of the outage to the CNO registry operator. The expected length of this outage is unknown. We are closely monitoring this situation, and will continue to issue updates as new information becomes available. We apologize for any inconvenience this may cause you.

Update – the registry is back online. -Chuckc

Mon Feb 11 15:01:03 PST 2002 — The web servers’ shared copy of Matt Wright’s FormMail has been upgraded and modified to reduce its potential as a conduit for spam. This change should be transparent to most users.

For more information please see news://news.sonic.net/sonic.help.www under the thread “FormMail update.” -Ian

Mon Feb 11 14:31:42 PST 2002 — The connection to our Oakland POP (this equipment serves all xxx-0174 numbers) has gone down, and we are working with Pacific Bell to troubleshoot the circuit. We don’t have an ETR yet, and alternate dialup numbers for most areas can be found via our POP finder: www.sonic.net/cgi-bin/pops.pl – We will advise with an ETR as soon as we have one. – Sonic.net Operations team Update: Repair work continues on the backhaul for the 0174 group. Pacific Bell’s engineers have tested the loop and believe that their portion is functional, and we have one of our OPS people en route to the Oakland facility to do on-site testing. Update: PacWest engineers have located the problem and repaired the downed line. The Oakland pop is now up and running and. If you were currently using a number ending in 0174 and were switched to another number, please switch back to your old number as soon as possible. Thank you for your patience. – Sonic.net Operations team

Sun Feb 10 10:24:54 PST 2002 — We are aware of the ‘All circuits are busy’ message on 707-522-1003 and we are working to resolve these issues as quickly as possible. – Steve

Fri Feb 8 16:33:01 PST 2002 — Tin has been rebuilt to include support for ispell, which was lost during shell.sonic.net’s last upgrade. -Ian

Welcome back, Ian! -Dane

Thu Feb 7 10:04:32 PST 2002 — A bug in our automated user management tools encountered a grievous error while running last night. It decided that while removing a user that it should, while it was at it, delete the entire ‘/home/l’ directory tree, removing all home directories for users starting with ‘l’. The bug has been squashed and all of the removed files and directories have been restored from our most recent backups. We are deeply sorry for any trouble that this has caused and have taken steps to assure that it does not happen again. -Kelsey

Update: This removal and recreation of the /home/l directory has caused problems with the server that handles ftp.sonic.net that can only be resolved by rebooting the server. ftp.sonic.net is being rebooted now and should be back up shortly. -Kelsey and Nathan

Thu Feb 7 18:04:08 PST 2002 — We have temporarily disabled telnet access to shell.sonic.net due to a vulnerability in the server. Patches are available and will be applied shortly. Once we are patched telnet service will be restored. SSH access is not affected. -Kelsey, Dane and Nathan.

Update: The telnet server on bolt has been patched and is now back online. -Kelsey