Month: June 2003

Thu Jun 26 10:26:57 PDT 2003 — Mail Relay Issues: popb4smtp was broken for a few hours this morning causing some users to receive relay denied messages while trying to send mail through our servers. Additionally, we also uncovered that users on our national roaming access network are not able to send mail to our new mail servers due to anti-spam measures in place. We are in contact with the vendor now to have the ACLs updated. -Kelsey, Nathan, and Matt.

Wed Jun 25 11:43:23 PDT 2003 — Network instability: Our pair of Cisco L3 switches that we use to provide network services to most of our servers have rebooted twice in the past hour. The crashes appear to be related to a BGP bug and unfortunately the bug appears to be tickled on both of them at the same time. We’re relocating what services we can to a different network segment to mitigate the affects of this until we are able to resolve the problems with the 3550’s. -Kelsey and Nathan

Tue Jun 24 22:00:10 PDT 2003 — Graton Rooftop Network Maintenance. Wednesday morning at 9:30am we will be doing network maintenance on the backhaul serving the Graton Rooftop wireless network. The interruption is expected to last less than 5 minutes. -Matt and Bryan

Tue Jun 24 16:59:37 PDT 2003 — Trouble with the SMS 1800: One of the ethernet uplinks from the SMS to our core network started showing errors at 6:30AM this morning. Since then it has flapped a few times and we are just replacing the cabling now to isolate the problem. This problem could be observed by brief loss of connectivity for DSL customers. Coincidentally at the same time that the interface dropped the last time, there also appears to have been fluctuations in the power grid in Santa Rosa– one of our carriers reported that some of their gear upstream from us rebooted and we also noted a brief drop. We are investigating to see if the two events could possibly be related. -Kelsey, Nathan and Russ.

Tue Jun 24 03:09:55 PDT 2003 — Night Operations Complete. All maintenance went as planned. No reboot of the SMS was required. Shell.sonic.net is running on the new switch, alot of patching has been replaced and alot of servers are running new kernels. -Kelsey, Nathan and Kevan

Mon Jun 23 12:24:23 PDT 2003 — Mail server upgrades: At 1:00PM we are migrating ‘mail.sonic.net’ to our new cluster. The new servers add support for STARTTLS, SSL encryption for SMTP. The primary reason for supporting STARTTLS is to allow for the encryption of username and passwords for customers that are using SMTPAUTH. It also has the advantage of encrypting mail while in transit to our servers and from our servers to remote SMTP servers that also support STARTTLS. We do not anticipate significant customer impact with this move, however it is possible that some mail clients may encounter trouble. If you have any trouble sending mail, please contact support. -Kelsey, Nathan and Russ

Mon Jun 23 12:16:19 PDT 2003 — Mail server problems: A configuration error caused our mail servers to briefly refuse mail from some of of our customers. Some users got error messages when trying to send Mail. No incoming mail was affected and no mail was refused without the sender receiving an error message. The problem was corrected within a few minutes – Russ

Mon Jun 23 19:59:56 PDT 2003 — Night Operations: Tonight, starting at 1:00AM, we will be working on our SMS 1800 in Santa Rosa as well as on some other routine projects. Most notably, we will be re-provisioning shell.sonic.net’s network to a different L3 distribution switch. Shell services will be briefly unavailable during the transition. -Kelsey, Nathan and Kevan

Fri Jun 20 12:27:55 PDT 2003 — FormMail now sends only to local addresses. To reduce its utility to spammers, the shared copy of FormMail installed on our webservers has been modified to only send to email addresses at Sonic.net hosted domains. If you need FormMail to send mail to an off-site address, please consider installing your own copy of the script. A secure FormMail replacement can be found at: nms-cgi.sourceforge.net/scripts.shtml

Wed Jun 18 15:40:01 PDT 2003 — SMS updates: At 1:00am Tuesday morning we will be performing updates to our Redback SMS 1800 in Santa Rosa. This unit serves both DSL and FRATM customers. Although the maintenance should not affect its operation, there is a possibility that we may be forced to reboot the device. -Kelsey and Nathan