Month: April 2004

Thu Apr 29 12:45:57 PDT 2004 — DSL maintenance outage May 3. We are going to upgrade one of our DSL trunks from SBC/ASI and the equipment that terminates it on Monday, May 3, 2004, at 12:01 AM. Customers who will be affected are the ones served from our San Francisco 200 Paul POP. The duration of the outage will vary by customer from 10 minutes to 2 hours as ASI reroutes our circuits from the old DS3 to the new OC3. -John and Nathan

Tue Apr 27 16:13:59 PDT 2004 — Graton Rooftop Outage. The Graton Rooftop network is offline. The problem appears to be with the head-end unit as other hosts at the tower have connectivity. We are working quickly to resolve the issue. -Matt and Bryan

Mon Apr 26 09:05:25 PDT 2004 — Occidental Rooftop Outage. An internal wiring issue has created a power outage in Occidental and caused our our head-end wireless router to go off-line. An electrician is on the way to fix the problem. There is no ETR at this time. -Matt

Update: Mon Apr 26 09:24:35 PDT 2004 — Power has been restored. Power has returned to the site and Rooftop service has been restored. -Matt

Thu Apr 22 02:52:32 PDT 2004 — Night Operations Complete. Most of the planned work was accomplished without event. sonic.sonic.net took a bit longer to get to reboot on the new kernel than expected and while we didn’t replace the gigabit switch we did shuffle around some of our layer three switches allow for expansion. -Kelsey, Nathan, Kevan and Jared

Wed Apr 21 12:01:34 PDT 2004 — Operating system upgrades across the board. We are upgrading kernels throughout our network do to a recent discovery of a kernel vulnerability with Linux. This is a “local root” vulnerability, which means we needed to upgrade all systems with customer access before posting about this in our MOTD. This has now been done. There were brief outages in single-server services, such as our shell server, as the systems were rebooted.

Administrators of Linux systems with multiple users may well be served by reading the following link:

www.securityfocus.com/archive/1/360810/2004-04-18/2004-04-24/0

-Scott, Geoff, and (for most of our systems!) Kelsey

Wed Apr 21 16:24:35 PDT 2004 — Night Operations. Tonight, starting at Midnight, we will be rebooting the remaining servers to finish upgrading the kernels. In addition to this, we and also going to be servicing our VPN concentrator and replacing one of our gigabit ethernet switches. Some services may be briefly affected while servers are rebooted or moved to the new switch. -Kelsey, Nathan, Kevan and Jared.

Wed Apr 21 16:07:14 PDT 2004 — Outbound Email Bulk Detection. We enabled outbound Email bulk this afternoon and the system is functioning as expected. If you send out broadcast email traffic and wish to be excepted from the detection, please contact support@sonic.net with an explanation detailing the type of mail you send, how frequently, to how many recipients and other bits of information that you feel are relevant. -Kelsey

Tue Apr 20 16:27:05 PDT 2004 — Spam Filtering Updates. The new default RBL set that we launched in conjunction with per-user MTA filtering controls have had a huge impact on the volume of spam that we accept for delivery to customers with little, or no, collateral damage. We’ve seen our mail flow to customers reduced from ~850 msg/min to less than 400 msg/min. This has significantly reduced load throughout our mail server complex. We didn’t stop here and are continuing to focus on additional methods to reduce the flood of spam.

On the 14th, we enabled a reflexive filter in the MTA that is rejecting between 110,000 to 180,000 known spams a day based off of signatures built off our own spam collection system in real-time. Due to limitations in the SMTP there is no way to make this filter optional. However, it is not needed as it is virtually impossible for a false positive hit to occur.

We are working to add some additional RBLs that users have requested, including some of the country based blackholes.us lists, and have a few other rules undergoing testing. Hopefully these will all be ready in the next few days.

We are also going to be enabling outbound email bulk detection in the next couple of days, possibly as early as tomorrow. The outbound bulk detection will begin to reject mail sent out through our servers once a unique or highly similar mail has been sent to more than a configured number of destination addresses. The bulk threshold is going to be initially set at 1000 and may change as need dictates. The outbound bulk detection goes a long way to ensure that our own outbound mail flow stays spam free and our mail servers stay out of blacklists when a customer’s computer or CGI is exploited to send spam through our network.

We are aware that our outbound bulk detection will block some legitimate uses of our mail servers. We strongly encourage customers who are running broadcast newsletters to set up and use a Sonic.net provided free mailing list which will not be so limited. Some users may be pleased to note that a mailman server is in the works to replace our majordomo sever. If using a free mailing list is not practical or possible, we can exempt customers from the bulk detection. For more information please contact support@sonic.net or visit news://news.sonic.net/sonic.antispam -Sonic.net Operations

Mon Apr 19 16:16:56 PDT 2004 — Occidental wireless service down. The Occidental Rooftop wireless service is offline. The tower is responding however, customers are not reachable. We are dispatching a technition with a new head-end unit to troubleshoot the problem. ETR is at least 1 hour. -Matt and Bryan

Sun Apr 18 22:04:54 PDT 2004 — SpamCop listing. For some time today both of our outbound mail servers were listed in bl.spamcop.net. Any mail sent by our servers to a remote server (including our own, where a user had bl.spamcop.net enabled) would have been bounced. The listing was in error and was related to an issue with the way SpamCop handles our known spam feed. SpamCop has assured us that this should not happen again. Unfortunately, due to DNS caching, even though our servers have been removed from the list it may be a day or two before all remote servers have forgotten about it. -Kelsey