UPDATE: We believe to have service restored to normal at this time. We are continuing to evaluate service and equipment.
We are currently seeing instability in our web cluster, we are investigating and hope to have the problem resolved shortly.
–SOC
Legacy DSL Maintenance
Tonight, September 13th, starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.
– Robbie and Tomoc
Update: The maintenance has been completed and all affected customers are back online.
Various system updates tonight
Tonight, 9/12/20012, starting at 11:00 PM, we will be pushing software and OS updates out to several system. These updates and associated reboots could cause intermittent issues using various member tools and other internal systems. No connectivity or voice services will be affected.
– William and the SOC
Fusion and FlexLink Long Distance Issue
We have identified an issue affecting a number of long distance and toll-free phone calls through our network this morning. Users would have noticed delayed ringing or busy signals on affected calls. We have routed around the affected trunks and calls should be completing normally now.
-Brian and the NOC
Brief webmail outage
This morning at approximately 10:00 AM a small number of customers may have experienced issues when connecting to our webmail cluster. One of the servers in our webmail cluster started experiencing issues which caused connections to be slower then normal and may have led to browser timeouts. Since connections to the problematic server were slow and not completely failing, our load balancer thought the server was still running correctly and didn’t pull it out of rotation for live traffic. Coincidentally the server in question was scheduled to be replaced today. We are also going to review the configuration of our load balancer so we can prevent problems like this from happening in the future. -William and the SOC
Update:
A small number of customers may have experienced trouble reaching our webmail services from approximately 2:30pm to 3:30pm. This was due to a configuration mismatch between old and new servers. System Operations has performed the necessary fixes to ensure better stability and capacity for the future.
— Joe and the SOC
Legacy DSL Maintenance
Tonight, September 10th, we will be performing maintenance on equipment serving legacy dsl customers in northern California. This maintenance window will be from 11:00pm – 2:00am. Expected downtime for affected customers is less than 15 minutes.
– Robbie and Tomoc
Update: Service has been restored to all affected customers as of 1:38AM.
Legacy DSL Outage
At 10:00pm, a port on one of our ATM aggregation switches failed, taking a small subset of our legacy DSL customers in the bay area offline. We are working on rerouting this traffic and bringing these customers back online.
-Tim and Tomoc
Update: Service has been restored to affected customers as of 10:32PM.
Legacy DSL Maintenance
Tonight, August 27th starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.
– Robbie and Tomoc
Router Maintenance
At 11:59pm on Saturday, August 25, we will be performing maintenance on equipment serving Fusion and Flexlink customers in the San Leandro area. Estimated down time is less than 20 minutes.
-Tomoc
IMAP & POP Server Upgrades
Tomorrow morning, in our ongoing email server upgrade project, we are starting the migration to our new IMAP and POP servers. This ends the year long beta and stability testing of a dovecot based cluster and we’re looking forward to be finished. While we do not anticipate any significant customer impact or that IMAP or POP services will be unavailable for any extended period, some hiccups are likely inevitable. Notably, some IMAP clients may completely re-download all messages when they sync with the new server.
If you have any questions or run into any issues checking your email over the weekend please check out http://forums.sonic.net or contact support for assistance.
Update: Sat Aug 25 18:57:41 PDT 2012. The migration has been completed and all IMAP and POP (and dependent services like webmail and voicemail) are running on the new cluster. Overall the migration went quite well but there were several short periods where some users may have encountered errors while we tuned the new cluster’s configuration for real-world load. If you experience any error or issues please let us know. -Kelsey