Tonight, this Tuesday, Sep 15 at 12:01 AM, we will be performing a software upgrade of the linecards that provide ADSL2+ service out of our Downtown Santa Rosa CO. This will cause a brief disruption of service to ADSL2+ customers in that area. Affected customers will experience approximately 5 minutes of downtime as the cards reboot onto the new software.
-Jared
Update: The ADSL2+ cards have all been upgraded to the new software and are up and functioning normally.
This evening, Saturday September 12, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from a heavily loaded DSL aggregation router to a newer, less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes. We apologize for the short lead time on this announcement, but we feel it is better to address the load issue before the weekend. Thank you for your understanding.
-Jared
Update: The migration has been completed without incident and all affected customers appear to be up and running normally at this time.
Update (01:10) : Mail migration done; should be faster accessing e-mail for those of you that were migrated. -Augie
This Saturday (12, September) at 12:01 AM we will be performing the last service impacting e-mail migrations from our existing e-mail storage infrastructure to our new (and greatly improved) storage system.
When it is completed this migration will mark a substantial improvement in our e-mail systems performance and availability for all of our customers.
During the maintenance window (12:01am – 1:00am) all customer e-mail systems will be unavailable; this includes POP, IMAP, and Webmail.
No e-mail will be lost during this time. -Augie
This afternoon at 12:15 PM a huge DoS attack was aimed at a customer on our Sonic Telecom CLEC network. The DoS attack was large enough to destabilize routing in our CLEC network, so customers may have seen 10-15 minutes of spotty connectivity as we tracked down and blocked the DoS. At this time, the attack has been blocked and all services are functioning normally.
-Jared and Nathan
Update (0100) : E-mail maintenance done; total down time was around 28 minutes; no e-mail was lost; let us know if you see any problems. – Augie
This Saturday (05, September) at 12:01 AM we will be performing the second of four service impacting e-mail migrations from our existing e-mail storage infrastructure to our new (and greatly improved) storage system.
When it is completed this migration will mark a substantial improvement in our e-mail systems performance and availability for all of our customers.
During the maintenance window (12:01am – 1:00am) all customer e-mail systems will be unavailable; this includes POP, IMAP, and Webmail.
No e-mail will be lost during this time. -Augie
This morning at approximately 8 AM our Support staff became aware that one of our DHCP servers was not serving DHCP leases properly, in such a way that our monitoring systems were not able to detect the problem. Our Support team escalated the problem to the Operations staff and we were able to remedy the problem by rebooting the partially-locked server. Service was restored by approximately 8:28 AM. We apologize for this outage and will be looking into ways of extending our monitoring so this unusual type of failure will not be missed again.
-Jared and the Support team
Update: The DHCP server has begun to fail again, so we have removed it from service. DHCP should be functioning normally again. If you are having problems obtaining a DHCP lease, please try rebooting your modem and any computers or routers that are having problems.
A couple of issues showed up this morning that were not picked up by our monitoring systems or testing after the night-op. First, some users migrated to the new message stores were unable do use IMAP or webmail to access their e-mail. This problem was fixed shortly after Tech Support reported it this morning. Second, one the servers in our inbound mail cluster was left in the old configuration and delivered some new mail to the old message stores. This was also fixed shortly after Tech Support reported it and any misdirected email has been redelivered to the correct location. -Kelsey
For about an hour this afternoon SSL connections to our POP servers (pop.sonic.net) were unavailable; we have remedied the problem and are investigating the cause.
During this time non-SSL connections to pop.sonic.net still worked, and both SSL and non-SSL connections to our IMAP cluster (imap.sonic.net) were unaffected.
–Augie, Kelsey, and William.
Update : E-mail maintenance finished a bit ago, just cleaning up; total observed down time around 15 minutes. -Augie
This Saturday (29, August) at 12:01 AM we will be performing the first of four service impacting e-mail migrations from our existing e-mail storage infrastructure to our new (and greatly improved) storage system.
When it is completed this migration will mark a substantial improvement in our e-mail systems performance and availability for all of our customers.
During the maintenance window (12:01am – 1:00am) all customer e-mail systems will be un-available; this includes POP, IMAP, and Webmail.
–Augie
This Friday, August 21 at 12:01 AM we will be performing maintenance to upgrade an ATM aggregation circuit. We are doing this work to clear a capacity problem that the existing link faces during peak times. Sorry for the short notice.
-Jared and Nathan
Update: The migration is now complete. Downtime for affected customers was approximately 5 minutes, except for 4 folks that many have seen as much as 15 minutes worth of outage. -Nathan