Disaster Recovery

Understand when RAID 60 is overkill

When it comes to RAID 60, Scott Lowe advises IT pros not to have a one-size-fits-all mentality. He explains the potential benefits and downsides of using RAID 60.

In a recent TechRepublic column, I extolled the merits of RAID 50 and discussed how I think that, by borrowing certain aspects of both RAID 5 and RAID 10, RAID 50 hits a sweet spot between RAID 5 and RAID 10 by providing a good balance between capacity, performance, and reliability. My RAID 50 column generated a lot of comments, including a couple about RAID 60; specifically, it was mentioned that disks often fail in groups, so from a data protection standpoint, why not go with RAID 60 to further protect your data? I'll provide a brief overview of RAID 60 and then examine both sides of the RAID 60 equation.

How RAID 60 works

The diagram in Figure A gives you an idea of how a RAID 60 array is constructed. Like RAID 50, RAID 60 is a multi-level disk set; you start with a bunch of RAID 6 sets, and then these sets are aggregated at a higher level into a RAID 0 array that has no redundancy on its own. However, each RAID 60 set does have redundancy and can withstand the loss of up to two disks in each parity set. In theory, in the diagram below, you could lose six of the 12 total disks (two in each set) and still have an operational array. As soon as you lose more than two disks in a single parity set, though, your world would come crashing down, as the RAID 0 set breaks, and you're back to data recovery mode.

Figure A

RAID 60 diagram

An additional note: RAID 60 requires a minimum of eight disks in order to operate -- you need at least two four disk RAID 6 sets to comprise a single RAID 60 set.

RAID 60 benefits and drawbacks

I like RAID 50 a lot, so you might think that I'm a big believer in taking that to the next level and that I would like RAID 60 even more; to be honest, I'm not sure. I see RAID 50 as a great balance between capacity, performance, and reliability, and I see RAID 60 as potentially imbalanced on the capacity side (to the negative) in order to support the increased reliability inherent in RAID 60.

I don't think you should simply avoid RAID 60 at all costs; instead, make the decision on a case by case basis with an understanding of the tradeoffs that you'll face. In fact, you might find that RAID 60 is a great fit when you need higher usable capacity and better reliability and can trade a little in write performance for it.

With RAID 60, you're going to lose anywhere from around 12% to 50% of your usable space to parity information. This is not a bad thing, and the whole design of RAID 6 is built around the idea that using more space (two disk's worth to be exact) to enhance reliability is a good thing. If you're ultra-concerned about reliability, are you more likely to use fewer disks per individual RAID 6 set? If so, this would decrease the overall usable capacity of the solution. In fact, in the diagram above, you'd lose 50% of your disk space to parity, so why not just go with RAID 10 in that scenario?

With RAID 6, you will take a performance hit (more so than with RAID 50) when it comes to writes, but reads will be boosted, as is the case with RAID 10 and RAID 50. The exact performance hit you take with writes under RAID 60 is largely dependent on the quality of your RAID controller and on what you're doing. If you're considering implementing a RAID 60 that eats 50% of your space in overhead, it's time to consider just using RAID 10, which will provide similar read performance and better overall write performance and provide similar levels of redundancy.

From a pure reliability perspective, a RAID 60 array is orders of magnitude more reliable than even RAID 50 arrays due largely to the extra parity disk employed in RAID 60.

The more disks you add to each individual RAID 6 set in a RAID 60 array, the higher percentage of usable space you get from the overall RAID 60 array. Perhaps the biggest tradeoff in RAID 60 is that you can build larger individual RAID 60 sets in a safer manner than is possible under RAID 50, so from that perspective, perhaps you can get more safely usable space from a RAID 60 array.

Conclusion

When it comes to RAID 60, I don't think IT pros should have a one-size-fits-all mentality. And before you jump on the RAID 60 train, be aware that there are potential downsides for usable space and performance that need to be considered, so choose wisely. For more information, I recommend checking out IBM's article and chart about selecting a RAID level.

Want to keep up with Scott Lowe's posts on TechRepublic?

About

Since 1994, Scott Lowe has been providing technology solutions to a variety of organizations. After spending 10 years in multiple CIO roles, Scott is now an independent consultant, blogger, author, owner of The 1610 Group, and a Senior IT Executive w...

12 comments
jakesty
jakesty

Read this link of information and you decide. http://www.miracleas.com/BAARF/BAARF2.html I believe a simple RAID 10 with a hot spare will give you just as much security as a RAID 5 or Raid 6 array but with substantially faster access. Join BAARF-Battle Against Any RAID Five/4/3

joshua.melton.ctr
joshua.melton.ctr

Has anyone considered RAID 55? A RAID 5 striped across three or four RAID 5 stacks? Yes, lots of loss, but lots of fault tolerance. - joking, though probably viable... Just remember: RAID 50 is fine! It's only ideal in conjunction with backups. Disk loss is inevitable and you have to be ready to handle that.

neilb
neilb

and the speed with which a replacement for a failed disk can rebuild. Given a reasonably swift new disk turnaround the only danger period for any RAID 5 set, whether or not in RAID 50, is during the rebuild of the new disk. If another disk in the same RAID 5 set fails before this is complete then your complete array is toast. So, calculate the risk and decide for yourself whether that risk is supportable. I've been using RAID 6 on large arrays of 1TB SATA disks in my SAN for some time because of that perceived problem. One day, I worked out the real numbers and I reckon that I'm wasting that extra parity disk. I haven't done a proper risk analysis on RAID 60 because I don't use it but, maybe I'll get the calculator out...

Chimel
Chimel

Kevaburg, I beg you to reconsider: The RAID 60 data parity space is the same as for RAID 6, it is the size of your whole RAID 6 array, minus 2 disks per RAID 6 array. So the "wasted" space is 50% only if you use 4-disk arrays. You should of course aim for RAID 6 arrays of 16 disks each. Considering groups of 16 2TB disks, the extra cost compared to RAID 5/50 is only 6%, or 2TB out of 32TB, $200 out of $3,200 (16 x $200 2TB disks.) The benefits of this extra 6% cost are immense: You can lose up to 2 disks in the same RAID 6 array without losing any data. You would lose the whole 32TB with RAID 5/50. If you take into account that multiple incidents in the same array are not rare, and that the HUGE time required to rebuild a single 2TB hot spare disk increases a lot the possibility for a second disk to go wrong (it was far less risky with smaller, say, 256GB disks), it is a no-brainer that RAID 60 is the best solution around. My own needs are not mission-critical, it's just video streaming for family and friends, but I can't even start to imagine spending several weeks or months re-ripping hundreds of DVDs. Even for write operations, I think I'd probably increase the write performance not by choosing RAID 50, but by selecting SATA III 2TB enterprise disks with big fat cache such as the Barracudas XT and SATA III RAID controllers when Adapter has them. Or use SSD disks for even higher performance (and if I were a company and we were in 2012. ;-) I have a 8 x 1.5TB RAID 6 array at the moment, but it's already full, so I plan on using RAID 60 with 2 groups of 16 x 2TB disks to start with. Reliability is much more important to me than sheer performance, or I wouldn't be using RAID in the first place. 6% strikes me as an acceptable cost and a good balance of real "Redundant" Arrays of "Inexpensive" Disks.

Neon Samurai
Neon Samurai

I've tended towards RAID1 or RAID10 and always using hardware raid due to losses (though losing personal data is a great way to learn about not loosing business data). I've been happy to give up half my usable space in exchange for mirrored drives. If one drive bakes, I pop it out and leave the machine running on the good drive while the fresh drive is braught back into sync. Perhaps I should take more time to develop trust in RAID5 based arrays. I just can't get around the feeling that it's less protection for my data though.

kevaburg
kevaburg

Thanks for a great article! It is interesting to know that there are so many different fault-tolerant solutions out there. But I do take issue with what seems to be overkill even for information that could be deemed to be mission-critical. Considering the loss of space (50%) in a RAID 60 configuration, plus the performance implications of parity caluculations, then surely RAID 5+0 is a better option. But then, if redundancy can be carried over to a second physical appliance and performance is a critical issue, then surely RAID 0+1 is better still from a performance perspective. Sure, 50% of the total available space is lost to redundancy (a necessary sacrifice) but it fulfills the needs of most enterprises in terms of redundancy and to an extend, high-availability. Sometimes I think there are people simply looking at ways to consume resources without any real tangible benefits.

Chimel
Chimel

What looks like a waste when all is fine and dandy will pay off when you have 2 disks failing. After all, you waste only $100 and 1TB per array (6% of raw disk space on an optimized array). Compare that with the cost of rebuilding your whole array from the backup. Probabilities of 2 disk failures may be low, but that extra RAID 6 security works basically the same way as an insurance policy. In my home usage case, backups are not even an option because of the sheer size of the data storage, so my only option is to rely on RAID 6. I'd probably still waste an extra disk with RAID 5/50, not for parity data, but as a hotspare.

jhoward
jhoward

Data integrity and performance have been the main reasons we migrated away from RAID 5/6 since we got burned so many times. With RAID 5 and the huge size of hard drives the "degraded" array was too degraded performance wise for too long for our needs while rebuilding. This is where RAID 10 shines. I will tell you that the cost of drives is insignificant vs the time and effort wasted on help desk calls and late night trips to the data center. I also have to point out that Linux software RAID (server hardware is cheap) was the deal maker for us on this. A 24 port RAID card is 10x more expensive than a non-RAID HBA plugged into a port multiplier backplane. Without this the cost of hardware RAID cards would be the largest prohibitive cost in our case. For fun check this out: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ That being said - if a situation exists where data integrity and performance are not your top 2 concerns then I would say consider RAID 50/60. Try pulling a drive in the middle of the day to see if your users can deal with the performance hit of a failed drive without blowing up your help desk. If they can then I would say RAID 50/60 is a decent choice.

neilb
neilb

It doesn't "look" like a waste, statistically it IS a waste and with around 30TB of SATA disks, a little more than 100 USD. :) I do have backups of everything and I have most of the stuff that matters mirrored onto another SAN that's around ten miles away.

kevaburg
kevaburg

The amount of time it takes to regenerate the parity data in a RAID5 array from a single failed disk is sometimes so prohibitive that it generates more helpdesk calls than if the data simply wasn't available! RAID10 is for me the best option and RAID60, although allowing for the loss of two disks in one array at one time, does not provide such high benefits for me to consider a change from my current strategy.

Chimel
Chimel

It's a waste statistically, but it's still only 1 disk per array, so a one-time cost of $200 or $400 if your config is 2 or 4 arrays of 15 or 8 disks. And then $15-$30 of power per year for the 2-4 extra disks. So, financially, it IS a good deal. It all depends on how critical and costly the disruption is, compared to running in degraded state. With RAID 5, as soon as a second disk in the same array fails, you lost the whole array and need to restore the backup and rebuild the whole array. Sure, this might never happen, but that's what risk management is about, create countermeasures to avoid getting in that situation in the first place, and also to put in place solutions to mitigate the risk and control the damages. No manager will ever refuse spending an extra $1K or 2 if explained in that context, it's their ass on the line. I guess you must be looking up the upcoming 3TB Seagate disks, you'd need only 10 to consolidate your whole SAN into a single array! ^-^

Editor's Picks