Brief webmail outage

This morning at approximately 10:00 AM a small number of customers may have experienced issues when connecting to our webmail cluster. One of the servers in our webmail cluster started experiencing issues which caused connections to be slower then normal and may have led to browser timeouts. Since connections to the problematic server were slow and not completely failing, our load balancer thought the server was still running correctly and didn’t pull it out of rotation for live traffic. Coincidentally the server in question was scheduled to be replaced today. We are also going to review the configuration of our load balancer so we can prevent problems like this from happening in the future.  -William and the SOC

 

Update:

A small number of customers may have experienced trouble reaching our webmail services from approximately 2:30pm to 3:30pm. This was due to a configuration mismatch between old and new servers. System Operations has performed the necessary fixes to ensure better stability and capacity for the future.

— Joe and the SOC

Legacy DSL Maintenance

Tonight, September 10th, we will be performing maintenance on equipment serving legacy dsl customers in northern California. This maintenance window will be from 11:00pm – 2:00am. Expected downtime for affected customers is less than 15 minutes.

– Robbie and Tomoc

 

Update: Service has been restored to all affected customers as of 1:38AM.

Legacy DSL Outage

At 10:00pm, a port on one of our ATM aggregation switches failed, taking a small subset of our legacy DSL customers in the bay area offline. We are working on rerouting this traffic and bringing these customers back online.

-Tim and Tomoc

Update: Service has been restored to affected customers as of 10:32PM.

Legacy DSL Maintenance

Tonight, August 27th starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Router Maintenance

At 11:59pm on Saturday, August 25, we will be performing maintenance on equipment serving Fusion and Flexlink customers in the San Leandro area. Estimated down time is less than 20 minutes.

-Tomoc

IMAP & POP Server Upgrades

Tomorrow morning, in our ongoing email server upgrade project, we are starting the migration to our new IMAP and POP servers.  This ends the year long beta and stability testing of a dovecot based cluster and we’re looking forward to be finished.  While we do not anticipate any significant customer impact or that IMAP or POP services will be unavailable for any extended period, some hiccups are likely inevitable.  Notably, some IMAP clients may completely re-download all messages when they sync with the new server.

If you have any questions or run into any issues checking your email over the weekend please check out http://forums.sonic.net or contact support for assistance.

Update: Sat Aug 25 18:57:41 PDT 2012.  The migration has been completed and all IMAP and POP (and dependent services like webmail and voicemail) are running on the new cluster.  Overall the migration went quite well but there were several short periods where some users may have encountered errors while we tuned the new cluster’s configuration for real-world load.  If you experience any error or issues please let us know.  -Kelsey

Legacy DSL Maintenance

Tonight, August 21st starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the San Francisco bay area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Anonymous Call Rejection

We deployed support for Anonymous Call Rejection in our voice network and management tools this morning.  When enabled, ACR prevents calls from being received where the calling party has blocked their outbound caller-id.  Users can enable or disable ACR in the voice membertools at https://members.sonic.net/voice/settings/ or by dialing *77 and *88.  For more information please see https://wiki.sonic.net/wiki/Anonymous_call_rejection and https://forums.sonic.net/viewtopic.php?f=14&t=821

International Calling Outage

We are presently experiencing issues with international calling on our voice network. We believe to have identified the source of the issue and are working with the vendor of the impacted circuit to restore service as quickly as possible.  An update will follow when we have confirmed resolution of the outage.

-Matt and Nathan

Update: Service has been fully restored as of 10:30am.

Legacy DSL Maintenance

Tonight, August 16th starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Update: The maintenance has been completed. Downtime for affected customers was around 5 minutes.