Author: admin

Thu Aug 1 11:50:42 PDT 2002 — Our SQL server experienced intermittent problems this morning. This caused Member Tools to be briefly inaccessible. No data was lost and all normal services are now restored. -Chris, Kelsey, Eli

Thu Aug 1 10:40:49 PDT 2002 — Broadlink move preparation. Broadlink will be doing some prep work on their wireless backhaul today at 11am. In order to move the backhaul link between 300 B Street and their network they will be moving customers to a temporary 10Mb wireless link. This will allow them to move the 100 Mb link that currently carries backhaul traffic. The move will not cause service interruption, however, peak usage on the current link is 7 Mb/s and the interim 10 Mb link will create a temporary ceiling for traffic bursts. -Sonic.net and Broadlink

Wed Jul 31 04:23:05 PDT 2002 — We brought the additional 75GB disk in typhoon live today, and gained another 60+ gigs of usable spool. This additional spool will help increase our binary retention as well as reducing IO contention on the other disks improving overall performance. -Kelsey and Nathan

Wed Jul 31 14:05:31 PDT 2002 — Broadlink will be rebooting their head end switch at 11:00 PM tonight. This will result in a few minutes of interruption for broadlink customers. – Russ

Wed Jul 31 13:18:13 PDT 2002 — Our Pac West (530-xxx-0174) numbers started returning ‘All Circuits Busy’ messages about 30 minutes ago, and we’ve tracked to an issue with the Telco. Pac West’s engineers have not given us an ETR, but I will keep this space updated. — Eli, Stephanie

Update: The problem is more widespread than initially perceived, and affects all of our xxx-0174 numbers served from Stockton. This is a good portion of Northern California (excluding the Bay Area), and we’re working with Pac West to get this repaired as quickly as possible.

Update, 15:05hrs: The problem was caused by an administrative fumble at Pac West. Our backhaul circuit was mistaken for another, and disconnected. Service is fully restored, and we’re in deep dialogue with Pac West. — Eli

Mon Jul 29 21:00:35 PDT 2002 — We will be upgrading sonic.sonic.net, one of our core administrative servers tonight with new faster CPUs and additional RAM. We are also going to be performing some non-intrusive reconfiguration in our core network. We may, time allowing, also try to bring up an additional 75GB of spool up on news.sonic.net. We will start with sonic at Midnight. -Kelsey and Nathan

Fri Jul 26 09:42:00 PDT 2002 — One of our frame relay circuits is experiencing problems. This is affecting a few of our FR customers. We are working with PacBell to get this issue resolved and hope to have it resolved shortly. – Steve and Matt K.

Update: The frame relay T1 came back up by itself, most likely caused by a something in PacBell’s network.

Thu Jul 25 13:40:09 PDT 2002 — Our SLB mail cluster is still exhibiting odd behavior. We are seeing poor NFS performance, and as a result, we are also seeing delayed email. It is also possible that we may be refusing inbound SMTP connections as the load on the servers climb as ‘dirty’ process pile up. We are currently investigating the cause of this problem and hope to have it resolved shortly. -Kelsey and Nathan

Update: The problem appears to have resolved itself but we expect that in reality it will start up again tomorrow as load increases. We were not able to find anything wrong with our configuration and will continue to debug and troubleshoot once it starts again.

Update: The problem did not return today. We’ll continue to investigate the circumstances which cause our servers to get into trouble and to exhibit poor NFS performance. At this time we believe that the problems may have been caused by a remarkably aggressive Rumplestiltskin Attack.

Tue Jul 23 13:23:48 PDT 2002 — shell.sonic.net (aka bolt) was not permitting dialup shell logins from our focal gear (numbers ending in 9811). After some investigation we found that the gear was on a new IP block that had not been authorized to rlogin into the shell server. We have added the IP block to the authorized IP list and all is working now. -Steve

Mon Jul 22 16:44:59 PDT 2002 — Our SLB web and mail server clusters has been exhibiting some odd behavior today. For short periods of time services are going unavailable do to some network issue. We are working to track down this problem as quickly as possible. -Kelsey and Nathan

Update: We tracked the problems down to the Ether Channel links going through our Alteon AD3 hardware load balancer. We are unclear why this was causing problems as it is a technology that we’ve been using for some time. We disabled all but one of the four trunked connections on both sides and everything is working fine.