Call Center Inbound Calls Down

The two PRIs feeding the customer support call center are currently down.  We are working with our PRI vendor to restore service.  We have narrowed it down to a physical issue, but do not have an ETA to restoration of service.   We will update this issue as they become available.

Currently our  turn around time for email support (support@sonic.net) is remarkably quick.

Update: Service has been restored.  The remote terminal serving our location lost AC power.  It is now on generator back-up.

Update: As of 5:40pm, the two PRIs serving our call center appear to be down again. While we work to expedite a second repair, our customer service representatives are still available at support@sonic.net .

 

Update: At 6:08 service was once again restored and we are taking incoming calls.  We will continue to monitor this situation and update any changes in status.

 

– Sonic.net Senior Support

 

 

Mail Server Upgrades

Thursday morning we’re going to replace the mail server cluster that handles delivery to customer mail spools.  For all but a handful of users that are using procmail to filter their mail on our servers this upgrade is expected to be completely seamless.  This upgrade will eliminate the queue delay problems that have shown up a few times over the last couple of months.  For more information please see this forum post.

Update:  The upgrades have been completed and everything appears to be working correctly at this time.  Mail delivery latency is back under our target of less than 1 second.

Inbound Email Delays and Pending Upgrades

This morning a portion of our inbound email flows ended up queuing on our MX edge servers due to excessive load on the internal servers that handle email delivery and filtering to local destinations  All queued email was delivered when the issue resolved itself on its own by 9AM.  We are currently tracking down how our monitoring failed to alert us to the problem.  Coincidentally, we’ve been working on several upgrades to our email server clusters.  We’ve recently replaced all of our spam filtering servers and already have hardware on order to replace the cluster that was responsible for the delays this morning.

Invasive Router Maintenance

Tonight, Wednesday, August 1, at 12:01AM we will be performing invasive maintenance on equipment serving Fusion and FlexLink products in San Bruno. Customer down time is expected to be less than 20 minutes.

-Sonic.net NOC

LA area legacy DSL maintenance

This Friday morning, at 12:00AM on July 27th, we will be performing intrusive maintenance on DSL aggregation equipment serving customers in the LA area. Expected downtime is less than 15 minutes.

Update: The maintenance was completed successfully and all affected customers are back online.

– Robbie and Tomoc

Legacy DSL Outage

A port on one of our ATM aggregation switches appears to have failed and has taken a small subset of legacy DSL customers in the bay area offline. We are working on rerouting this traffic and bringing these customers back online ASAP.

-Robbie and Tomoc

Update: All affected customers have been moved to a new port and should be back online. If you are still experiencing trouble with your DSL connection please call our support department as usual.

Legacy DSL Maintenance in LA

This Friday morning, at 12:00AM on July 20th, we will be performing intrusive maintenance on DSL aggregation equipment serving customers in the LA area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Network Maintenance

This coming Monday, July 16th, from 9:00PM to 3:00AM, we will be performing major intrusive work within one of our core points of presence in San Jose. We do not expect any service impact to our customers as a result of this operation. Fusion voice users may notice a very short delay in their calls when initially dialing out during a portion of this window.

– NOC

Update: Maintenance completed without a hitch, customers should not have experienced any service interruption.

Sonic.net Router Maintenance

Tonight, July 8th, at 12:00am, we will be performing maintenance on routers that aggregate Business-T and Frame-Relay connections.  The procedure will require a reboot of the routers. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

 

Update: As some of you may have noticed, 12:00am tonight is actually July 6th. I apologize for any confusion this may have caused.

Update 2: The router maintenance has been completed and all affected customers are now be back online. Total downtime was about 3 minutes.

Sonic Telecom Emergency Backbone Maintenance

Due to a hardware failure in the Sonic Telecom backbone we will need to perform emergency router maintenance over the next hour. We do not believe this work will have any impact on customer traffic but there is the potential for a brief service interruption. Further updates will follow shortly.

-Tim and Nathan

Update: Maintenance work has been completed. There should have been no impact to customer traffic. This evening, from 12:00AM until 1:00AM, we will be performing further maintenance of a similar nature. Again we do not expect any impact to customer traffic but there is a possibility of brief service interruptions during this time.