Friday, January 02, 2009

A Storage tragedy.

Imagine if your company, which survived the downturn in the market in 2008, was shut down because no one backed up the data which was the bread and butter of your company. It looks like a company has experienced this nightmare and is now going out of business.

JournalSpace lost all of its customer's data and is out of business. I was tipped off by a friend who sent me this article.

The Dangers of Casual Information Backup
JournalSpace, which has been around for almost six years now as a prominent blogger website, is done. Due to a catastrophic data loss, either due to software error or malicious intent, the website managers have lost the entire contents of the blogs stored upon it. Alexa places JournalSpace at a rank of about 100,000 among websites, and with nearly 14,000 viewers per month, JournalSpace was a relatively high-traffic website. The data loss is impossible to correct and JournalSpace is effectively destroyed.

On a technical side, what occurred is still unclear. JournalSpace maintains their server on what is known as a RAID configuration, where everything written to one hard drive is 'mirrored' to the other drive in a duplicate image. So if the main storage unit breaks down, all the data is stored on another duplicate and it's an easy enough job for an individual to simply switch over to the working drive. Most any computer can be configured for RAID with a minimum of effort.

JournalSpace had taken no other effort to back up their data other than constantly mirroring the drives. The damage that occurred then was not physical, but rather based in software. Whether due to a catastrophic operating system failure (OS X) or malicious intent, the contents of both drives have been completely overwritten. Akin to reformatting your hard drive, this means that the data is completely irretrievable.



Below is the content of the website and all it says now.

**************

Tuesday:

Journalspace is no more.

DriveSavers called today to inform me that the data was unrecoverable.

Here is what happened: the server which held the journalspace data had two large drives in a RAID configuration. As data is written (such as saving an item to the database), it's automatically copied to both drives, as a backup mechanism.

The value of such a setup is that if one drive fails, the server keeps running, using the remaining drive. Since the remaining drive has a copy of the data on the other drive, the data is intact. The administrator simply replaces the drive that's gone bad, and the server is back to operating with two redundant drives.

But that's not what happened here. There was no hardware failure. Both drives are operating fine; DriveSavers had no problem in making images of the drives. The data was simply gone. Overwritten.

The data server had only one purpose: maintaining the journalspace database. There were no other web sites or processes running on the server, and it would be impossible for a software bug in journalspace to overwrite the drives, sector by sector.

The list of potential causes for this disaster is a short one. It includes a catastrophic failure by the operating system (OS X Server, in case you're interested), or a deliberate effort. A disgruntled member of the Lagomorphics team sabotaged some key servers several months ago after he was caught stealing from the company; as awful as the thought is, we can't rule out the possibility of additional sabotage.

But, clearly, we failed to take the steps to prevent this from happening. And for that we are very sorry.

So, after nearly six years, journalspace is no more.

If you haven't yet, visit Dorrie's Fun Forum; it's operated by a long-time journalspace member. If you're continuing your blog elsewhere, you can post the URL there so people can keep up with you.

We're considering releasing the journalspace source code to the open source community. We may also sell the journalspace domain and trademarks. Follow us on twitter at twitter.com/jsupgrades for news.

*******

Every company has to judge what the value of a backup is, and what strategy they are going to use to maintain their backup. How much insurance is too much? What is your D/R plan?

No comments: