Month: June 2014

Connectivity Issues to Santa Rosa Datacenter

This morning, beginning around 3:45 AM, a host within our Santa Rosa datacenter became the target of a very large DDoS attack. We are currently in the process of battling this attack and users may notice reachability problems to services such as our email servers and member tools during this time.

– Robbie

Legacy DSL Outage

Update( 7:49AM): This issue has been resolved.

A portion of legacy DSL customers in Northern California are currently offline due to an issue with a piece of dsl aggregation gear. We are working on diagnosing and repairing the equipment asap.

– Robbie

Fusion/FlexLink Intrusive Maintenance

Update( 12:20AM ): This maintenance is now complete.

This evening, beginning at 11:59PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in  San Jose. Expected downtime is less than 15 minutes. A small portion of customers may need to reboot their DSL modems after these code upgrades take place.

– Robbie

Non-Intrusive Network Maintenance – Bay Area

6/19/14 – 12:42AM PDT – This Maintenance is now complete.

Tonight beginning at 11:59PM PDT we will be performing non-intrusive maintenance to equipment serving Fusion and Flexlink circuits in the bay area. This is a follow-up to the maintenance last night. No service impacts are expected.

-Tim J.

cdr maintenance

This maintenance is now complete. Everything should be functioning as normal.

Tonight (6-17-14) beginning at 8:00 pm we will be performing maintenance on our cdr server. Expected downtime is approximately 4 hours. During this time call detail records will be unavailable to view via our member tools.

-William

Non-Intrusive Bay Area Network Maintenance

Update: 6/18/2014 1:45AM – This maintenance is now complete. Users may have noticed slight (sub 5%) packet loss for brief instances during this migration, but things should be back to normal now.

Tomorrow night (6/17/2014) beginning at 11:59PM PDT, we will be migrating traffic to newly upgraded portions of our Bay Area network. This covers all Fusion and Flexlink customers in the Bay area. No customer impact is expected during this maintenance.

-Tim J.

Fusion/FlexLink Intrusive Maintenance

Update (12:35AM): This maintenance is now complete.

This evening, beginning at 11:59PM, we will be performing intrusive maintenance on equipment serving a portion of Fusion and FlexLink customers in  San Francisco. Expected downtime is less than 15 minutes. A small portion of customers may need to reboot their DSL modems after these code upgrades take place.

– Robbie

Spam Filtering and MX Cluster Upgrades

Over the past several months we’ve watched a marked increase in the spam making its way through our filtering.  And we’re not alone.  Spam volume has increased over 6 times since October 2013 globally and many spammers have become increasingly sophisticated in thwarting filters, explicitly testing and tuning delivery through common systems to ensure their spam reaches their targets.  In response, we’ve completed several upgrades that have had a big impact reducing spam in our inboxes.

We finished a migration to our next generation MX servers last week.  The new MX servers feature a completely rewritten custom policy enforcement engine.  Like most MX filters, it makes heavy use of a collection of reputation services in conjunction with policies like rate limits and checks for basic compliance.  There’s one important difference with the new system: it only allows customers to opt-out of the filtering where our old MX filters allowed for some per user customization.  When a customer opts out, we’ll make our “best effort” at excluding a customer’s inbox from any filtering.

We’ve also made a couple of  important changes to our SpamAssassin installation including the integration of a commercial plugin.  This, along with a redoubled effort to maintain our own set of custom rules, has significantly increased its accuracy. We hope these changes have made as big an impact on your inbox as they have on ours.

Customers may opt-out of the new MX filters and adjust their SpamAssassin preferences using the Spam Configuration membertool.  Please join us in the forums if you have any questions.

-Kelsey

Fusion/FlexLink Intrusive Maintenance

Update (12:30AM): This maintenance is now complete.

This evening, beginning at 11:59PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in the Sacramento area. Expected downtime is less than 15 minutes. A small portion of customers may need to reboot their DSL modems after these code upgrades take place.

– Robbie

list.sonic.net upgrade.

Our list server has been upgraded with new hardware and a newer version of the mailman list software. Most importantly, the new dmarc_moderation_action has been set to “Munge From” for all lists on the server. This change will allow the server to rewrite the from header when sending messages to domains that would reject the email due to strict DMARC policies. This change will be most noticed by list users with yahoo.com and aol.com email addresses. I have started a forum post here for discussion: https://forums.sonic.net/viewtopic.php?f=5&t=2139 . Please let us know if you have any problems.

 

-Grant and System Operations