New sonic.net webmail application

We are pleased to announce that we’re ready for users to test our new webmail application.  It features a modern interface that utilizes ajax and html5 and supports features like drag-and-drop and has a robust plugin system that allows us to extend its functionality.  All of our existing webmail applications will be replaced by the new one on April 22nd.  We encourage users to start using it now – it really is better in just about every way!

For now it is available at https://webmail-beta.sonic.net

Please let us know if you have any questions or feedback in the forums.

 

Emergency ATM Maintenance

We are performing an emergency reload on equipment serving a subset of legacy DSL, Business-T, and FRATM customers in the Bay Area and Modesto/Stockton area. We will send another update shortly once the maintenance has completed.

 

-Tim

 

Update: Maintenance has been completed and all affected customers should be back online.

Sonic Telecom Call Completion Issues

As of around 11:20PM PST we began experiencing issues with an upstream voice provider for our telecom network. We have confirmed it is an identified issue that they are working to resolve, and have taken steps to reroute our own network to mitigate customer impact.  For the duration of this issue, customers may still experience intermittent inbound calling success.  Further update will be provided upon outage resolution.

-Matt, Nathan and Tim

Update 1:48AM – Our upstream reports that the issue is resolved, and we’re seeing the same. -Nathan

DDoS Distruption

This morning, at around 9:15AM, a DDoS attack towards Sonic.net owned equipment de-stabilized part of our hosting network resulting in the inability to access to the www.sonic.net website and some others. The issue has been handled and access to the sites restored as of 9:45AM.

– Noc and Soc

Emergency Legacy DSL Maintenance

At 4:15pm, a small subset of legacy DSL subscriber connections became unstable. We are investigating the cause of this issues and are working to stabilize the affected circuits as fast as possible.

Update: All affected subscribers have been migrated to new equipment. Affected customers may have noticed several service interruptions lasting no more than 3 minutes each. Please note that under some circumstances, rebooting the DSL equipment on site may be required.

-Tomoc and Robbie

Ongoing DNS Server DoS Attack

Over the past few days we’ve seen a massive increase in both the number and volume of DNS Amplification Attacks using our recursive name servers.  This is likely due to the fact that our new name servers provide more verbose answers and are therefore amplify traffic more effectively than our old servers.  We unfortunately had to roll back blocking off-net use of our recursive servers and blocking these requests entirely is not currently an option at this time.  To mitigate the effects of the attacks both on our systems and their targets, we’ve instituted rate limits on the total number of queries per second any given IP address is able to source to our servers.  The rate limits are high enough that they should not interfere with any normal (and acceptable) use.  However, it is possible that a customer doing bulk DNS lookups (such as log processing or running a busy mail server) may run into issues and experience intermittent delays resolving host names.

-Kelsey, Augie and Nathan

DNS Service interruption.

DNS service was interrupted to our ns1.sonic.net name server cluster at 5pm PST for approximately 10 minutes.

All service has been restored and we are investigating the root cause of the problem.

–Augie

Update: Another brief service interruption today ( 21 March ) at 4:31pm PST for approximately 7 minutes as we were updating our name servers. Service was restored at 4:38pm PST.

Emergency ATM maintenance

We are performing an emergency reload on equipment serving a subset of legacy DSL, Business-T, and FRATM customers in the Bay Area and Modesto/Stockton area. We will send another update shortly once the maintenance has completed.

 

-Tim and the NOC

 

Update: Service for affected customers should now be restored. At around 3:03PM, a controller card in one of our aggregation switches performed an automatic failover to the standby card. For unknown reasons, the standby card was not operating correctly once it took over, causing an outage for some percentage of subscribers. We were forced to perform a full reset of the device to restore service. We are investigating both the cause of the failover as well as the additional issues experienced afterwards and we apologize for the duration of this outage.

ATM Maintenance

This evening, beginning at 11:00PM, we will be performing invasive maintenance on equipment serving a subset of ATM subscribers in the Bay Area and Sacramento. This will affect Business-T, FRATM, and legacy DSL services specifically. We expect downtime for the affected customers to be less than 15 minutes.

 

-Tim, Robbie, and Tomoc

Update: The maintenance work has been completed as planned. All affected customers should be back online.

Legacy DSL Maintenance

Tonight, March 7, starting at 11:30pm, we will be performing maintenance on equipment serving a small subset of legacy DSL customers in the Bay Area. Expected customer down time is less than 30 minutes.

Update: Maintenance complete. Affected customers may need to restart their DSL equipment.

-Tomoc