Thu Apr 17 12:21:00 PDT 2008 — News Server Outage. news.sonic.net is currently offline due to back to back disk failures in the redundant load balanced pair of servers that handle news readers services. At this time it is unclear how long it will take to restore news reader services.
Failure One: Last night nnrp1 (the news reader slave server) was taken off-line due to a disk failure. Users may have seen poor news performance if their streams were on this cluster until it was taken off-line. The failed disk was replaced this morning with a spare and work was started to rebuild it and bring it back into service. Failure Two: The news server systems have removable face plates that cover all of the drive bays. In order to confirm the correct configuration of the drives in nnrp1, the faceplate for nnrp0 was removed to expose the hot swap disk carriers. At this point, nnrp0 (the now single, and master news reader server) unexpectedly powered off. Upon reboot, one of it’s news related disks has also hard failed.
We will continue efforts to restore news services as quickly as possible. In any case, it will take at least the remainder of the day to restore services and is likely that article numbering will be lost. -Kelsey
Update:: Thu Apr 17 15:31:16 PDT 2008 — We have scavenged parts from nnrp1 to rebuild nnrp0 and are currently restoring a backup that we hope will allow us to retain article numbering across all but our local sonic.* groups. Unfortunately, it is going to take several more hours before the data restoration is complete. If this fails, we will bring the server back up with all groups starting at zero. -Kelsey
Update:: Thu Apr 17 21:21:35 PDT 2008 — news.sonic.net is back online. We were able to restore nnrp0 with article numbering more or less intact but did lose numbering of our local groups as expected. At this time our spoolers and extra header feeds and busy pushing their backlogged header feeds in. Time will tell how big of a gap remains once it has caught up. Some users may find that they have to unsubscribe/subscribe from our local sonic.* news groups in order to see new news or force their news readers to do a full refresh of the group’s headers. -Kelsey