This afternoon at 12:15 PM a huge DoS attack was aimed at a customer on our Sonic Telecom CLEC network. The DoS attack was large enough to destabilize routing in our CLEC network, so customers may have seen 10-15 minutes of spotty connectivity as we tracked down and blocked the DoS. At this time, the attack has been blocked and all services are functioning normally.
-Jared and Nathan
Update (0100) : E-mail maintenance done; total down time was around 28 minutes; no e-mail was lost; let us know if you see any problems. – Augie
This Saturday (05, September) at 12:01 AM we will be performing the second of four service impacting e-mail migrations from our existing e-mail storage infrastructure to our new (and greatly improved) storage system.
When it is completed this migration will mark a substantial improvement in our e-mail systems performance and availability for all of our customers.
During the maintenance window (12:01am – 1:00am) all customer e-mail systems will be unavailable; this includes POP, IMAP, and Webmail.
No e-mail will be lost during this time. -Augie
This morning at approximately 8 AM our Support staff became aware that one of our DHCP servers was not serving DHCP leases properly, in such a way that our monitoring systems were not able to detect the problem. Our Support team escalated the problem to the Operations staff and we were able to remedy the problem by rebooting the partially-locked server. Service was restored by approximately 8:28 AM. We apologize for this outage and will be looking into ways of extending our monitoring so this unusual type of failure will not be missed again.
-Jared and the Support team
Update: The DHCP server has begun to fail again, so we have removed it from service. DHCP should be functioning normally again. If you are having problems obtaining a DHCP lease, please try rebooting your modem and any computers or routers that are having problems.
A couple of issues showed up this morning that were not picked up by our monitoring systems or testing after the night-op. First, some users migrated to the new message stores were unable do use IMAP or webmail to access their e-mail. This problem was fixed shortly after Tech Support reported it this morning. Second, one the servers in our inbound mail cluster was left in the old configuration and delivered some new mail to the old message stores. This was also fixed shortly after Tech Support reported it and any misdirected email has been redelivered to the correct location. -Kelsey
For about an hour this afternoon SSL connections to our POP servers (pop.sonic.net) were unavailable; we have remedied the problem and are investigating the cause.
During this time non-SSL connections to pop.sonic.net still worked, and both SSL and non-SSL connections to our IMAP cluster (imap.sonic.net) were unaffected.
–Augie, Kelsey, and William.
Update : E-mail maintenance finished a bit ago, just cleaning up; total observed down time around 15 minutes. -Augie
This Saturday (29, August) at 12:01 AM we will be performing the first of four service impacting e-mail migrations from our existing e-mail storage infrastructure to our new (and greatly improved) storage system.
When it is completed this migration will mark a substantial improvement in our e-mail systems performance and availability for all of our customers.
During the maintenance window (12:01am – 1:00am) all customer e-mail systems will be un-available; this includes POP, IMAP, and Webmail.
–Augie
This Friday, August 21 at 12:01 AM we will be performing maintenance to upgrade an ATM aggregation circuit. We are doing this work to clear a capacity problem that the existing link faces during peak times. Sorry for the short notice.
-Jared and Nathan
Update: The migration is now complete. Downtime for affected customers was approximately 5 minutes, except for 4 folks that many have seen as much as 15 minutes worth of outage. -Nathan
This Wednesday, August 19 at 12:01 AM we will be performing fiber maintenance on one of our OC12 circuits that aggregates DSL service in the Bay Area. We will be moving the OC12 to a new in-building cross-connect. Expected downtime is less than 5 minutes as we move the fiber from one cross-connect to another. This maintenance will affect some DSL and Business-T customers in the Bay Area.
-Jared and Nathan
Update: This maintenance is now complete. Actual downtime was approximately 4 minutes and 40 seconds.
We upgraded the kernel running on ‘bolt’ (our customer shell server) today to address a potential security issue.
This required a reboot of the system. The system is backup and running smoothly.
-William
We have completed the migration to our new web hosting security configuration. The new system improves our already robust web site hosting platform with fully compartmentalized security and eliminates the need for cgiwrap configurations. For more information please see news://news.sonic.net/sonic.help.www -Kelsey and Nathan