catastrophic failure [updated: Recovery!]

Share/Save/Bookmark

admin
by admin
12-Nov-2009
 

[UPDATE: Nov. 12, 7:30AM Pacific]

We were able to recover the month-long lost data and are now working to restore it on a new server. We will be down as far as new user contributions for a few hours, but will be back with full functionality later today. JJ is working on transferring all the new submission that happened in the past day or so. However, as we warned before, most comments (made in the last day) will be lost.

Backround

So, for a while now (pretty much since the elections) we've been trying to move to a different hosting setup. We use Rackspace which has a very high reputation for service and support in the industry. Rackspace has a new cloud service called Mosso. We moved there earlier this summer, but due to a large variety of technical problems, it didn't work out for us. So we moved back to regular Rackspace hosted server.

When we moved back, about a month ago, our new Rackspace service told us they can't support backup operations until we switch to a new operating system (Redhat from CentOS) which we were scheduled to do about 2 weeks from now. But until then, we needed to do backups.

Earlier this week, another member of our team asked RS to add an external hard drive so we can have backups onto there until our new server is ready. When they went to add this hard drive, which was supposed to happen without any service interruptions, they reported that the existing drives had failed and recovery attempts were not successful. It's such a rare thing (for a Raid 5 system to have two hardware failures in minutes) that one person involved has already theorized that it might have been done on purpose. While we can't draw any conclusions at the moment, we are asking questions and pursuing the matter.

While Rackspace is working on trying to get the data off the old drives, which could take days. We are stuck with the last successful backup we did have, which was from October 10.

I hope I didn't give you a headache. JJ is ready to stay up all night trying to update the page with whatever he can. Please don't email him! It won't help. He will probably write in this blog and you can respond to him (soon.)

Our last updated page is cached at google.

Thank You for your continued patience and understanding.

Share/Save/Bookmark

 
Natalia Alvarado-Alvarez

kheyli sepaas! :o)

by Natalia Alvarado-Alvarez on

 

ناتاليا


sbglobe

Incremental backup – just a suggestion

by sbglobe on

I know that taking “full Volume backup” takes a long time on daily bases and therefore not doable. I work with (IBM) Z Enterprise Servers (aka mainframes) and we take “incremental backup” every night. The software is “intelligent” enough to only back up the new or change items (we then take full volume backup weekly). Of course we could be working with different requirements mandated by nature of our business.

Best wishes for future “hardware crash” as you know it will happened again (when you least expect it)     


capt_ayhab

Mr.'s Javid, Admin Foad and whomever it may concern

by capt_ayhab on

Thank you gentlemen. 

-YT