Month: November 2003

Sun Nov 16 09:07:25 PST 2003 — Some Broadlink problems persist. Some lingering problems persist on Broadlink’s backhaul link. This has been traced to problems with the link radios’ firmware. Broadlink has been working with the vendor, with no resolution yet — in the interim, Broadlink will be replacing the backhaul link radios with those made by another vendor. They estimate the new link will be up by noon today. -Scott, and Tim from Broadlink

Update: Sun Nov 16 14:02:50 PST 2003 — Broadlink problems persist. Broadlink is currently on premises working to realign the new wireless backhaul equipment. ETR one hour. -Kavan, Matt, and Tim from Broadlink

Update: Mon Nov 17 00:48:48 PST 2003 — Broadlink problems fixed. After many hours of troubleshooting, the issues have been resolved. Many thanks to Tim, Jason, Scott and Chris for their long hours and good work. Customers having issues at this time should reboot their CPE device. If problems persist, please call technical support. -Matt, Tim, Jason and Scott W.

Fri Nov 14 14:31:14 PST 2003 — Broadlink site came up…then went back down. Broadlink had solved the problem with their backhaul link, but then lost a switch at the facility. Broadlink is working hard to restore operation of the site. -Scott, and Linda from Broadlink

Update: ETR is approximately 3:30pm. -Scott and Linda

Update: Fri Nov 14 16:43:47 PST 2003 The repair team is still on site, we will update the MOTD as soon as we know more. -Scott and Linda

Update: Fri Nov 14 20:02:44 PST 2003 The backhaul link and supporting equipment are back online. Jason and Tim have worked hard over the past 24 hours to find and fix the various problems that caused the outages. Everyone is keeping a close eye on the network and things have been stable for the past couple of hours. -Matt, Tim and Jason

Update: Sat Nov 15 18:57:33 PST 2003 Broadlink Wireless links are back up. Some network recovery issues are preventing a small number of customers from coming back on line. Broadlink is continuing to work on these individual problems. Currently we have no ETR. — Scott, Tim, Kavan, and Jason

Thu Nov 13 20:44:37 PST 2003 — Backhaul link between Sonic’s fiber facilities and one of BroadLink’s primary wireless POPs has gone down. Customers served from most of BroadLink’s network in central, west, and north east Santa Rosa are affected. J. Kane is presently working on resolving at site. Still evaluating for ETA service restoration. – Tim

Tue Nov 11 16:19:49 PST 2003 — Changes to Anti-Spam Services: We’ve discontinued two old services that used to be part of our spam countermeasures. First, users used to be able to send spam to spam@sonic.net which would remail the spam examples into the sonic.spamcan news group for review by other users and staff. As of yesterday, we are no longer supporting this; it’s hasn’t proved useful for us to continue to collect spam in this fashion. Our thanks go out to everyone that has helped us in the past by bouncing their spam to this address. Second, a reminder that no-spam@sonic.net, which used to provide a friendly auto-response explaining that our customer did not want to receive spam, no longer does so. Mail to no-spam@sonic.net bounces with a standard ‘user unknown’ DSN. -Kelsey

Wed Nov 5 10:48:55 PST 2003 — System username database propagation problem. We just corrected a problem with the propagation of username info to Sonic.net servers. This means username adds, deletes, and changes just started working — they had stopped working at about 11:30am yesterday. We apologize for the inconvenience, and will make it better: we are adding some (more) intelligence to the propagation process to notify us of this particular problem. -Scott

Wed Nov 5 15:42:46 PST 2003 — Broadlink outage. Broadlink is experiencing problems with their head end system, affecting all Broadlink customers. No estimate of a repair time yet. -Broadlink and Sonic Operations

Update: Wed Nov 5 16:24:00 PST 2003 — Service has been restored.

Tue Nov 4 14:36:42 PST 2003 — Load Balancer Issues: We’ve just uncovered that one of our Alteon AD3 load balancing switches is apparently corrupting ethernet frames off of at least one of it’s port with single bit errors. These errors were going completely undetected by the servers or switches; the corrupted frames have the correct checksum information.

The single-bit error corruption in Ethernet frames on this switch was resulting in the transposition of characters in email streams sent to and from the affected servers. For example, the letter ‘A’ might have been translated to the symbol ‘~’, or ‘.’ to ‘x’. In most cases, the errors introduced would go unnoticed — they’d appear to be typos. However, attachments that were corrupted could be rendered unusable and it’s also possible that errors at certain points, or those which introduced certain control characters, could have caused fatal errors.

We are in contact with our vendor to identify if the problem is a hardware or software fault in the switch. We’ve temporarily worked around the corruption by disabling the affected servers. Once we’ve gathered sufficient debugging information, we’ll swap to the standby Alteon which is not exhibiting the problem and re-enable the affected servers. -Kelsey and Nathan

Mon Nov 3 16:39:07 PST 2003 — National Roaming Access Changes: We shutdown access to our old national provider this afternoon. Any roaming users will need to update their settings to use the new access numbers which are available at www.sonic.net/cgi-bin/pops.pl The new provider gives us increased coverage area and better overall service. -Sonic Operations

Mon Nov 3 15:08:51 PST 2003 — Sendmail Upgrades: We made a small change to the sendmail binaries in use on our mail cluster to resolve some infrequent STARTTLS related errors. Normally, this upgrade would have gone without notice. However, the new binaries didn’t have the proper permissions set when they were installed. This didn’t affect normal email flow. However, a small subset of users who use procmail to forward their mail to other addresses off of our servers may find that their mail was not get forwarded until the problem was noticed and corrected. -Kelsey and Eli

Sat Nov 1 16:34:56 PST 2003 — Router maintenance Tuesday, 11/4/03. We are scheduling the replacement of some router hardware at the Focal POP in SF at 5 am. We anticipate a 15 minute outage. This will affect dialup access through that POP, as well as some general routing instability as our network routes around the loss and then reconnection to UUNet. -John and Nathan