Month: August 2012

At 11:59pm on Saturday, August 25, we will be performing maintenance on equipment serving Fusion and Flexlink customers in the San Leandro area. Estimated down time is less than 20 minutes.

-Tomoc

Tomorrow morning, in our ongoing email server upgrade project, we are starting the migration to our new IMAP and POP servers.  This ends the year long beta and stability testing of a dovecot based cluster and we’re looking forward to be finished.  While we do not anticipate any significant customer impact or that IMAP or POP services will be unavailable for any extended period, some hiccups are likely inevitable.  Notably, some IMAP clients may completely re-download all messages when they sync with the new server.

If you have any questions or run into any issues checking your email over the weekend please check out http://forums.sonic.net or contact support for assistance.

Update: Sat Aug 25 18:57:41 PDT 2012.  The migration has been completed and all IMAP and POP (and dependent services like webmail and voicemail) are running on the new cluster.  Overall the migration went quite well but there were several short periods where some users may have encountered errors while we tuned the new cluster’s configuration for real-world load.  If you experience any error or issues please let us know.  -Kelsey

Tonight, August 21st starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the San Francisco bay area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

We deployed support for Anonymous Call Rejection in our voice network and management tools this morning.  When enabled, ACR prevents calls from being received where the calling party has blocked their outbound caller-id.  Users can enable or disable ACR in the voice membertools at https://members.sonic.net/voice/settings/ or by dialing *77 and *88.  For more information please see https://wiki.sonic.net/wiki/Anonymous_call_rejection and https://forums.sonic.net/viewtopic.php?f=14&t=821

Tonight, August 16th starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Update: The maintenance has been completed. Downtime for affected customers was around 5 minutes.

The two PRIs feeding the customer support call center are currently down.  We are working with our PRI vendor to restore service.  We have narrowed it down to a physical issue, but do not have an ETA to restoration of service.   We will update this issue as they become available.

Currently our  turn around time for email support (support@sonic.net) is remarkably quick.

Update: Service has been restored.  The remote terminal serving our location lost AC power.  It is now on generator back-up.

Update: As of 5:40pm, the two PRIs serving our call center appear to be down again. While we work to expedite a second repair, our customer service representatives are still available at support@sonic.net .

Update: At 6:08 service was once again restored and we are taking incoming calls.  We will continue to monitor this situation and update any changes in status.

– Sonic.net Senior Support

Thursday morning we’re going to replace the mail server cluster that handles delivery to customer mail spools.  For all but a handful of users that are using procmail to filter their mail on our servers this upgrade is expected to be completely seamless.  This upgrade will eliminate the queue delay problems that have shown up a few times over the last couple of months.  For more information please see this forum post.

Update:  The upgrades have been completed and everything appears to be working correctly at this time.  Mail delivery latency is back under our target of less than 1 second.

This morning a portion of our inbound email flows ended up queuing on our MX edge servers due to excessive load on the internal servers that handle email delivery and filtering to local destinations  All queued email was delivered when the issue resolved itself on its own by 9AM.  We are currently tracking down how our monitoring failed to alert us to the problem.  Coincidentally, we’ve been working on several upgrades to our email server clusters.  We’ve recently replaced all of our spam filtering servers and already have hardware on order to replace the cluster that was responsible for the delays this morning.