Sun May 20 04:37:33 PDT 2007 — Scheduled maintenance of customer data is complete.
Customer /home and FTP data moved quickly and the bulk of the work which was the Web data took the bulk of the time, but was completed within the scheduled maintenance window.
All services have been restored and all customer data has been verified. -Augie and Kelsey
Fri May 18 06:37:23 PDT 2007 — Dialup service interruption. A hardware failure of one of our routers in our San Francisco POP caused a loss of connectivity to some of our dialup users. We have routed around the failed hardware and all affected customers should have full connectivity again. We are working to replace the failed hardware ASAP. -Jared and Nathan
Update Fri May 18 07:07:09 PDT 2007 — In order to completely resolve the failed hardware problem, we needed to reload the affected router. This would have temporarily affected connectivity to a number of T1 and dialup customers. We will continue to investigate this failure. At this time, everything is functioning normally. -Jared
Tue May 15 15:50:40 PDT 2007 — Scheduled maintenance of our customer data storage system during the early hours of Sunday May 20th.
Early Sunday morning we will be migrating away from our current set of NetApp Filers to a new set of NetApp Filers. These new Filers feature faster disks, more RAM, greater speed and reliability as well as a much higher storage capacity.
This upgrade marks a significant improvement to the system that houses the majority of our customers’ data, and will allow Sonic.net to provide continued performance and storage capacity far into the future.
Over the last few weeks we have been transparently migrating customer data on to the new NetApps and have been very impressed by their performance and reliability.
The total scheduled maintenance window is 4 hours (0:00 – 04:00), but we expect the actual time spent migrating data to be much less. During this time services such as shell, mail, and websites will be unavailable for a period of time as we migrate that particular data set.
-Augie, Kelsey, Dan, Nathan, Dane.
Wed May 9 11:04:45 PDT 2007 — DSL hardware restoration. The DSL hardware that failed yesterday has been repaired and at 12:01 AM Thursday May 10, we will be moving back to the original hardware. Expected impact is 5 minutes of downtime for the DSL customers terminated on this hardware. -Nathan and Jared
Update Thu May 10 00:14:50 PDT 2007 — Technical considerations have caused us to postpone this migration until 1:00 AM this evening. Sorry for the delay! -Nathan
Tue May 8 10:57:02 PDT 2007 — A hardware failure in one of our DSL gateway routers has caused packet loss and slow performance for a percentage of our DSL customers. We have identified the problem and expect service to be fully restored in 30 minutes. -Eli, Operations Update Tue May 8 11:21:40 PDT 2007 — We have replaced the failed equipment, and service has been restored.
Mon May 7 14:16:09 PDT 2007 — DSL migration. At 12:01am on Wednesday, May 7th we will be upgrading our DSL aggregation circuit in the Modesto/Stockton area. Customers in that region will experience an outage of approximately 15 minutes as their circuits are re-routed to the new link. -Nathan and Kelsey
Update Tue May 8 11:34:58 PDT 2007 — This migration has been re-scheduled to 12:01am on Thursday, May 10th. -Nathan
Update Wed May 9 16:04:10 PDT 2007 — Continued problems with some SONET equipment have caused this migration to be re-scheduled yet again. It will likely happen early next week. -Nathan
Sat May 5 17:13:08 PDT 2007 — DSL interruption. At approximately 5:00 PM today, some of our DSL customers experienced a 5-minute interruption of service. The interruption was found to be caused by a brief traffic flood, possibly a DoS attack. The flood stopped shortly after it began, and service was restored. We are currently investigating the source of the traffic flood. -Jared and Nathan