Month: August 2005

Wed Aug 31 16:53:35 PDT 2005 — Shell Server Policy Change. Shell server users are now allowed access to cron on shell.sonic.net. Please use the service appropriately; cron jobs running too frequently or otherwise using excessive system resources will be disabled. -Kelsey and Scott

Thu Aug 25 12:32:32 PDT 2005 — Phone system problems. We are having intermittent problems with our phone system causing us to drop calls; we are working to resolve the problem. -Tech Support

Update Fri Aug 26 10:13:33 PDT 2005: Phone system problems persist. We are working with our phone system and telco providers to resolve the dropped-call issue. We have no ETR at this time, but no Internet services are affected. -John F and Jasper C

Update: The phone system issues have been resolved. -Tech Support

Tue Aug 23 16:04:11 PDT 2005 — www.sonic.net downtime. www.sonic.net was down for approximately nine minutes around 3pm today when a feature upgrade to our web cluster went awry. -Augie

Fri Aug 19 15:36:07 PDT 2005 — LATA 9 DSL Issues. We have been tracking a capacity problem within our provider’s Stockton network that manifests as light latency and packet loss to some destinations. Only DSL customers in the Stockton area are affected. Our vendor has isolated the trouble and is working to repair the problem. While they have not provided us with an ETR, we expect normal service to be restored within the next couple of days. -Network Operations

Thu Aug 18 18:14:28 PDT 2005 — Customer Router Maintenance. On Saturday, 8/20 @ 12:01AM we will be performing maintenance on one of our customer routers at our 200 Paul Ave. POP in San Francisco. This maintenance will cause a brief loss of connectivity for T1 and dialup customers served from this facility. We expect service to be fully restored within a few minutes following the maintenance. -Network Operations

Tue Aug 16 15:42:28 PDT 2005 — Internal SQL server outage: Our primary internal SQL server that handles the storage for tools is down for emergency maintenance to replace faulty hardware. This affects membertools and many of the systems used by tech support. Services will be restored as soon as possible. -Operations

Tue Aug 16 14:27:51 PDT 2005 — Shell services on shell.sonic.net had some problems related to the user database. This manifested as trouble logging in to the shell, and as problems with unix userid which would be observed as errors in some applications.

The trouble came and went for a few minutes at a time over the course of 20 minutes or so while we worked to isolate and resolve the bug. Sorry for the late MOTD! -Dane & team

Sat Aug 13 16:12:45 PDT 2005 — Our secure web server was not serving traffic from approximately 5:15 to 5:45 this morning. Cause was traced to memory overconsumption due to processes not correctly freeing unused memory. We will be investigating further to prevent this from happening again. -Jared

Thu Aug 11 12:14:23 PDT 2005 — Shell server ssh upgrade. We’ve upgraded ssh and sshd on our shell servers to the latest version of OpenSSH with localizations for our environment. (Yea!) -Kelsey and Nathan

Update Thu Aug 11 20:43:08 PDT 2005: The problems with the ‘bot’ monitor on the shell server were noted early this afternoon – it was disabled pending further debbuging. -Ops

Tue Aug 9 23:30:32 PDT 2005 — Mail server storage failure. One of our Netapp filers that is used for mail spools went down and its partner filer has failed to take over its duties. We’re en route to fix the issue. Some users will notice inbound message delays until this is fixed, but no mail will be lost. -Nathan, Kelsey and Jared

Update Wed Aug 10 00:04:16 PDT 2005: The failure has been cleared. We’re currently determining the cause to ensure it doesn’t happen again. -Nathan and Kelsey