Month: January 2005

Sat Jan 29 10:14:28 PST 2005 — User Database Issue. Between 8:54 and 9:13 this morning it appears the system that creates and manages the unix passwd databases encountered an unexplained error and pushed a truncated file to our servers. The next automated update corrected the situation. During this period mail to many sonic customers was returned with ‘User Unknown’ errors. Similarly, many users would have received errors when retrieving their mail, trying to access FTP, and in specific cases, some users’ websites may have returned errors. -Kelsey

Thu Jan 27 11:22:26 PST 2005 — New DSL Reverse DNS Tool. DSL customers with static IP addresses can now set their own reverse DNS records with the new member tool. The only requirement imposed on the RDNS records is that the forward already matches the requested reverse record. All manually configured and currently valid records were preserved. sonic.sonic.net/members/dsldns/ -Kelsey and Chris B.

Tue Jan 25 21:31:07 PST 2005 — Core4 FastEthernet failure. A FastEthernet interface in our Santa Rosa Core4 switch failed, which caused loss of connectivity with our old backbone network. The interface was replaced with an onsite spare, downtime was approximately 40 minutes.

A router used as part of our web server migration was on the old network, and the result was that our web cluster was unavailable during the outage. We are removing this single-point-of-failure as soon as possible. -Scott, Kelsey, and Nathan

Tue Jan 25 16:12:58 PST 2005 — Broadlink backhaul problem. Broadlink reports a backhaul circuit down between their Tmac and Rosenberg sites. Jason is working on a fix. No ETR yet… -Scott, and Jason from Broadlink

Update Tue Jan 25 17:17:38 PST 2005: Jason has repaired the backhaul by replacing a cable. The problem was weather-related. -Scott, and (thanks to) Jason of Broadlink

Mon Jan 24 15:30:34 PST 2005 — SMTP Authentication hiccough and port 25 blocking. Earlier today our POP before SMTP authentication system was accidentally disabled when installing and configuring a fresh compilation of sendmail to support SMTPS (SSMTP – SMTP over SSL on port 465) for mail clients that do not correctly support STARTTLS on port 587. We require password authentication when using SMTPS. Additionally, at some point in the future we will also require password authentication on port 587. At this time everything is working correctly despite a few bumps along the way.

NOTE: Many users connecting to our outbound mail servers from SBC/Yahoo DSL lines (not Sonic.net DSL lines) or Comcast cable Internet services may need to update their server settings to work around port 25 filters both of these providers appear to be enabling in their networks. -Kelsey

Wed Jan 19 23:33:01 PST 2005 — Border router reload. One of our border routers in Santa Rosa reloaded due to a memory error. The error was similar to the error experienced by one of our customer attach routers earlier in the day. These are definitely not software failures, though. This time, the Santa Rosa 2 POP was offline for around 4 minutes after the issue. The reload also might have caused spotty connectivity to our Focal dialup numbers for half a minute. We’re investigating causes, and will keep an eye open for any further trouble. As always, duplicate dial-up service was available for the briefly affected numbers. -Nathan

Wed Jan 19 19:50:44 PST 2005 — Customer router reload. One of our customer attach routers in Santa Rosa reloaded due to a memory error. This caused some T1s to experience approximately 4 minutes of unreachability. We are watching the router for further signs of trouble, and have full on-site spares in case a hardware swap is necessary. -Nathan and Kelsey

Tue Jan 18 17:59:50 PST 2005 — In order to provide better service to our DSL subscribers we are now monitoring all of our DSL subscriber circuits for performance and latency problems. All DSL subscribers can expect to see a very small flow of ICMP traffic from latency-tester.sr.sonic.net directed to their DSL attached systems. -Kelsey and Nathan

Fri Jan 14 12:15:05 PST 2005 — DSL Maintenance in Santa Rosa. At 12:01 AM Saturday morning are going to do some work on one of the SMS’s uplinks into our core network. We do not expect this to be a service impacting event but customers may see a short period of impacted performance. -Kelsey and Nathan

Thu Jan 13 08:47:40 PST 2005 — Broadlink TMAC tower packet loss. There is some packet loss on the Broadlink TMAC tower. Broadlink thinks it may be a bad cable — if that is the case, ETR is approximately 9am. -Scott, and Tim and Jason from Broadlink