Disaster Recovery

RAID 6 -- Do you really want it?


If you like RAID 5, you are almost certainly going to like RAID 6. If you have concerns about RAID 5, and if those concerns have to do with recovering from a disk failure, this article is not going to make you feel any better.

For several years now RAID 5 has been one of the most popular RAID implementations. Just about every vendor that supplies storage to enterprise data centers offers it, and in many cases it has become -- deservedly - a well-trusted tool. RAID 5 stripes parity and blocks of data across all disks in the RAID set. Even though users now must devote about 20% of their disk space to the parity stripe, and even though read performance for large blocks may be somewhat diminished and writes may be slower due to the calculations associated with the parity data, few managers have questioned RAID 5's usefulness.

There are however, two major drawbacks associated with using RAID 5. First, while it offers good data protection because it stripes parity information across all the discs within the RAID set, it also suffers hugely from the fact that should a single disc within the RAID set fail for any reason, the entire array becomes vulnerable -- lose a second disc before the first has been repaired and you lose all your data, irretrievably.

This leads directly to the second problem. Because RAID 5 offers no protection whatsoever once the first disc has died, IT managers using that technology have faced a classic Hobson's choice when they lose a disc in their array. The choices are these. Do they take the system off-line, making the data unavailable to the processes that require it? Do they rebuild the faulty drive while the disc is still online, imposing a painful performance hit on the processes that access it? Or, do they take a chance, hold their breath, and leave the drive in production until things slow down during the third shift when they can bring the system down and rebuild it without impacting too many users?

This choice, however, is not the problem, but the problem's symptom.

The parity calculations for RAID 5 are quite sophisticated and time consuming, and they must be completely redone when a disk is rebuilt. But it's not the sophistication of all that math that drags out the process, but the fact that when the disk is rebuilt, parity calculations must be made for every block on the disk, whether or not those blocks actually contained data before the problem occurred. In every sense, the disk is rebuilt from scratch.

An unfortunate and very dirty fact of life about RAID 5 is that if a RAID set contains, say, a billion sectors spread over the array, the demise of even a single sector means the whole array must be rebuilt. This wasn't much of a problem when disks were a few gigabytes in size. Obviously though, as disks get bigger more blocks must be accounted for and more calculations will be required. Unfortunately, using present technology RAID recovery speed is going to be constant irrespective of drive size, which means that rebuilds will get slower as drives get larger. Already that problem is becoming acute. With half-terabyte disks becoming increasingly common in the data center, and with the expected general availability of terabyte-sized disks this fall, the dilemma will only get worse.

The solution offered by most vendors is RAID 6.

The vendors would have you believe that RAID 6 is like RAID 5 on steroids: it eliminates RAID 5's major drawback - the inability to survive a second disk failure - by providing a second parity stripe. Using steroids of course comes with its own set of problems.

RAID 6 gives us a second parity stripe. The purpose of doing all of the extra math to support this dual parity is that the second parity stripe operates as a "redundancy" or high availability calculation, ensuring that even if the parity data on the bad disk is lost, the second parity stripe will be there to ensure the integrity of the RAID set. There can be no question that this works. Buyers should, however, question whether or not this added safety is worth the price.

Consider three issues. RAID 6 offers significant added protection, but let's also understand how it does what it does, and what the consequences are. RAID 6's parity calculations are entirely separate from the ones done for the RAID 5 stripe, and go on simultaneously with the RAID 5 parity calculations. This calculation does not protect the original parity stripe, but rather, creates a new one. It does nothing to protect against first disk failure.

Because calculations for this RAID 6 parity stripe are more complicated than are those for RAID 5, the workload for the processor on the RAID controller is actually somewhat more than double. How much of a problem that turns out to be will depend on the site and performance demands of the application being supported. In some cases the performance hit will be something sites will live with, however grudgingly. In other cases, the tolerance for slower write operations will be a lot lower. Buyers must balance the increased protection against the penalty of decreased performance.

Issue two has to do with the nature of RAID 5 and RAID 6 failures.

The most frequent cause of a RAID 5 failure is that a second disk in the RAID set fails during reconstruction of a failed drive. Most typically this will be due to either media error, device error, or operator error during the reconstruction - should that happen, the entire reconstruction fails. With RAID 6, after the first device fails the device is running as a RAID 5, deferring but not removing the problems associated with RAID 5. When it is time to do the rebuild, all the RAID 5 choices and rebuild penalties remain. While RAID 6 adds protection, it does nothing to alleviate the performance penalty imposed during those rebuilds.

Need a more concrete reason not to accept RAID 6 at face value as the panacea your vendor says it is?  Try this.

When writing a second parity stripe, we of course lose about the same amount of disk space as we did when writing the first (assuming the same number of disks are in each RAID group). This means that when implementing RAID 6, we are voluntarily reducing disk storage space to about 60% of purchased capacity (as opposed to 80% with RAID 5). The result: in order to meet anticipated data growth, in a RAID 6 environment we must always buy added hardware.

This is the point at which many readers will sit back in their chairs and say to themselves, "So what?  Disks are cheap!" And so they are -- which naturally is one of the reasons storage administrators like them so much. But what if my reader is not in storage administrator? What if the reader is a data center manager, or an MIS director, or a CIO, or a CFO? In other words, what if my reader is as interested in operational expenditures as in the CAPEX?

In this case, the story becomes significantly different. Nobody knows exactly what the relationship between CAPEX and OPEX is in IT, but a rule of thumb seems to be that when it comes to storage hardware the OPEX will be 4-8 times the cost of the equipment itself. As a result, everybody has an eye on the OPEX. And these days we all know that a significant part of operational expenditures derives from the line items associated with data center power and cooling.

I already find myself speaking to corporate and government data center managers that are experiencing "no build-out" mandates from their management. Because of the increasing expense of electricity, such sites are on notice that they will have to make do with what they already have when it comes to power consumption. Want to add some new hardware?  Fine, but make sure it is more efficient than whatever it replaces.

When it comes to storage, I'm quite sure that we will see a new metric take hold. In addition to existing metrics for throughput and dollars-per-gigabyte, watts-per-gigabyte is something on which buyers will place increased emphasis. That figure, and not the cost of the disk, will be a repetitive expense that managers will have to live with for the life of whatever hardware they buy.

If you're thinking of adding RAID 6 to your data protection mix, consider the down-stream costs as well as the product costs.

Does RAID 6 cure some problems? Sure, but it also creates others, and there are alternatives worth considering. One possibility is a multilevel RAID combining RAID 1 (mirroring) and RAID 0 (striped parity), usually called either RAID 10 or RAID 1+0. Another is the "non-traditional" RAID approach offered by vendors who build devices that protect data rather than disks. In such cases, RAID 5 and 6 would have no need for all those recalculations required for the unused parts of the disk during a rebuild.

Choose wisely.

(Mike Karp is a senior analyst with Boulder, Colo.-based Enterprise Management Associates (http://www.emausa.com/), an industry research firm focused on IT management. Mike can reached at mkarp@enterprisemanagement.com)

24 comments
ShaneDaly
ShaneDaly

Very interesting thread. I have an Areca 1882 card running 24 x 4TB HDD = 96TBRAID6


I need a very safe stable system hence dual disk redundancy but having read all the comments here I don't feel as safe as I did 30min ago!


Other than LTO which is slow what other backup method can I use to insure my data?  A separate JBOD?


Cheers


Shane

mlderr
mlderr

I just had an argument with a software vendor where he demanded a raid-5 over a raid-1. I pointed out that in either case if one drive fails-still running but, also in either case if more than one drive failed - dead server. I was thinking that a raid-1 would outperform a raid-5. I ended up with a raid-1 boot and a raid-5 for data with a hot spare. I think that raid-5 WAS the original way to get more storage when drives were so expensive. For a 25-computer doctors office, they are currently using 500gb (6-years of patient data) so, I thought 1TB velociraptors would be the ticket - mirrored. right or wrong??

james.c.siano
james.c.siano

Having a hot spare in any monolithic server or array is such a low cost, usually easy to implement in any configuration. Well worth the cost of an extra disk for any config...helps one sleep at night. I'd put in a hot spare even with a RAID 6.

CListo
CListo

Best alternative to this problem is RAID 0+1 But no all raid controllers support it

Richard Kirk
Richard Kirk

RAID10 is better! What is wrong with RAID10? - far better than RAID5 in every way - as disks are cheap the primary argument for using RAID5 (or 6) died in the late 90's!

Greybeard770
Greybeard770

Since the minimum for RAID 5 is 3 disks, does RAID 6 require 4 disks? My math skills calculate you are using 5 disks in your RAID array since you have 80% usability with RAID 5 and 60% with RAID 6. I think that would be 75% in RAID 5 and 50% using 4 disks. Hot spares sound better. Or maybe you need to be buying more dependable disks.

VBJackson
VBJackson

Your observation that going from RAID 5 to RAID 6 cuts your available capacity fro 80% down to 60% is a valid concern. However, you then seem to recommend using mirroring - which guarantees that you will only get 50% capacity. Even granting that rebuilding a mirror doesn't have nearly the performance hit of rebuilding a parity stripe your argument falls a little flat, especially when the later comments about OPEX are also added in.

BALTHOR
BALTHOR

This type of RAID is for companies like IBM or NCR.I think that whoever wrote RAID was surrounded by Military and they were patting him on the back.

dj4904
dj4904

Actually, you could sustain irretreivable data lose the moment you loose one member of a RAID10 array if during rebuild data from the remaining member of the original mirrored pair is unreadable. Only a dual-parity protection mechanism can sustain a single disk failure with protection from bad blocks during rebuild.

neilb
neilb

At the very least, I want the option then [b]I'll[/b] make the decisions as to the RAID level that I used based on the importance of the data. My current SAN has a mix of fibre channel and SATA and for 300GB fibre channel disks, RAID 5 is just fine. For arrays made up of 800GB SATA, the rebuild time onto the hot spare can be over a day if the SAN is being hit hard as rebuild is very processor intensive and is of a lower priority than data reads and writes. For some crucial data which is on the cheaper SATA, because it's big data and also not hit too hard, I want - and have - the safety net of RAID 6. I use RAID 5 on SATA for stuff that isn't so important. 800GB fibre-attached SATA disks aren't so cheap that I can justify getting the 5TB that I need from a 10-disk raid 1+0 set when I can use a 7-disk RAID 6 set instead.

arcangelo
arcangelo

I think with RAID 5 and Online Spare, you can have peace of mind if the RAID fails and also with this setup there is no need for RAID 6.

Florinel
Florinel

RAID 5E works like the descriptions in this article and I use it from years. It needs a minimum of 4 hard disks and survive with two defect disks.

dj4904
dj4904

It is that very paragraph that this author's article spiralled out of control into nothing more than worthless fear- mongering and shockformation. Facts: RAID5 incurs a single disks worth of added physical write penalty. RAID6 incurs two disks worth. For this reason, the physical write speed penalty for a RAID5 array is 50% of the speed of a single disk; we must simultaneously write to the active block disk, and to the parity block disk. The RAID6 array incurs a 66% write penalty, writing simultaneously to the active block disk, the first-level parity block disk, and the second-level parity block disk. This write speed penalty is mitigated by write-back caching on most controllers in a UPS- assured system but the maximum physical write speed cannot be overcome. If write performance required cannot sustain this penalty, and if high-availability is an absolute requirement, then multi-level RAID is the only option. Next, the arbitrary % disk loss figures quoted are absurd. You are correct that the author is using 5 disks, which is unheard of in the enterprise. The most common RAID5 implementation being purchased in the mid-level sector is the Apple XRaid shelf at 6 data drives and 1 drive for hot- spare (XRaid does not support RAID6). This physical layout incurs -as all RAID5 implentations using a hotspare- an N-2 space penalty and comes to 5/7, or 71.4%. RAID6, if even possible on the XRaid, would typically incur the exact same capacity penalty as RAID5 since a hot spare is not normally used (since the need to immediately begin rebuilding the array is not as critical since a second disk failure during rebuild (which is not uncommon) will not incur data loss. If however, we did want the exponentially-greater data protection offered by a hot- spare in a RAID6 implementation of 7 total disks, the capacity penalty would be N-3, or 4/7, or 57.1% Now let's look at the most common RAID6 implementations addressing simultaneously the power, datacenter space, raw capacity, and data protection... these are typically 12 to 16 member stripes with 83.3% to 87.5% capacity efficiency respectively. RAID5 would incur the same rates, but lets say we used no hotspare (gasp), the rates would be 91.6% to 93.8% with less data protection in a failure condition than RAID0. No thank you. Nexsan offers a product that currently can provide a 42- member RAID6 stripe SATA-based array with a raw capacity of N-2 where N=750GB (that's 30TB flat my friend). This is with a heat signature of 570W (single controller model) and 4U of rack space. That equates to 19W/TB (19mW/GB) and 7.5TB/U (7500GB/U). Raw numbers for this configuration show block-level write speeds of 128MB/s. The theoretical maximum speed of 2Gbps Fibrechannel is 250MB/s so the worst streaming write-speed penalty is approximately 50% over RAID10 (assuming RAID10 could saturate the Fibrechannel in this scenario, which is likely). Bottom line, is there is no Swiss Army Knife in storage. Everything is a comprimise. Protection, Performance, Cost... pick 2. As performance of drives today is at least double those of just 5 years ago (extremely conservative estimate) the penalty for RAID6 vs RAID5 is increasingly becoming an acceptible penalty for protection of very critical data, or (more likely) the SMB market that does not have the luxury of being able to sustain the complete loss of such large amounts of data for the time it would require to restore completely from backup. Business continuity is paramount to performance in the majority of this market. My personal opinion is that if your performace margin is so thin that you are forced to choose a RAID5 implementation over a RAID6, then you are not designing a sustainable or scalable system with the best tools available today and should consider a different physical storage model (such as RAID10 or RAID60) over putting data protection and business continuity at risk. One last comment regarding "you need to be buying more dependable disks", there are two kinds of people in the world: those that have had a hard drive crash... and those that will. -=dave [edit: fixed some typos]

shraven
shraven

I agree here. This piece needs some reworking and reposting. As stated, it's just plain wrong (20% capacity loss for RAID 5 - only with 5 disks! Less with more disks, duh), misleading (RAID 1+0 is worse on a lot of the factors the author brought up), and willfully obscure (CAPEX, OPEX? This isn't economics where we strive to use obscure terms for concepts everybody understands... this is a tech article so use the words, not acronyms for CIO, CFOs.) I especially hate authors who state a fixed % in overhead on RAID 5. It just shows they don't have a good understanding about RAID 5 to begin with, or are lazy in their writing(state your assumptions). Either way, do I really want to trust that person? Losing 1 disk on a 14 drive array is a negligible space hit, and only 7% loss!

jamesdtuttle
jamesdtuttle

At one previous employer, the cost of failure could potentially reach $15,750 per HOUR in an extreme case, and this wasn't that large a business. Any executive who sweats the cost of failsafe hardware and its trivial electrical demand is to my mind, an idiot. Our IT manager had older equipment as completely redundant servers, which could be brought online in short order. Was she paranoid, or wasteful ? Not to my mind. Very practical considering the cost of failure. I'd recommend her approach any day.

Why Me Worry??
Why Me Worry??

Oh wait, wrong type of RAID...LOL :-) Dual parity has been around for quite some time (sometimes referred to as data guarding by vendors like HP). I personally don't think it's needeed on servers per se, but it should be implemented on SAN arrays or NAS devices that host critical data.

fklein2
fklein2

While an online spare allows the rebuild to start right away, it does not allow for a second failure during the rebuild process. It also does not benefit any in space as it will result in same space available as raid6. With multi-terabyte drives arrays, the rebuild time is in the almost a day to rebuild time frame. The chance on losing a second drive with a multi-drive raid set is not only possible but can be probable when using 14 drives in a raid set.

neilb
neilb

What do you do if another disk in the array fails before the rebuild onto the on-line spare is complete? Start scrabbling for backups, that's what! The justification for RAID 6 is something that is becoming much significant as the capacity of disks goes on doubling whilst the access speed doesn't change. I leave you to think on how long it's going to take to rebuild a 1000GB hot standby in a RAID 5. You are TOTALLY vulnerable for that time.

Poordirtfarmer
Poordirtfarmer

Dave, Your 1st paragraph lost me. You seem to base your % on a 2 disk RAID 5. Wouldn't that be very unusual?

Tony K
Tony K

Yeah, the author assumes a 5-disk RAID array, but it's easy enough to ignore minor issues to look at the overall article content. Nitpicking is a worse habit than simply ignoring the minor details. You're ready to throw the whole article out (and the article does a good job giving an overview of RAID6) on a minor technicality and... ...your own ignorance. If you're working in IT and you don't know what capex and opex are, you really need to consider another field. IT isn't about technology, it's about meeting the needs of the business. If you don't understand that and those needs, you and your business are destined to fail. As an IT person you are REQUIRED to have an understanding of economics, business management, payroll, HR, accounting, manufacturing, etc, etc, etc. It's not about you getting the newest toys to play with on someone else's dime.

dj4904
dj4904

e x a c t l y -=dave

patrick
patrick

Having had experience with Multiple RAID environments from professional workstations running linux to terabyte arrays on Windows 2003 Servers and clustered across state lines (on fiber backbones ... and to you Brits it IS spelled fiber not fibre), such experience talks volumes to what actually works and what does not work in REAL LIFE practice. Raid 5 with a Hot spare IS THE BEST OPTION, PROVIDED you use multi-stage imaged solutions, ISO imaging with RIB/ILO setups virtualizing critical servers on the latest clustered virtual solutions. Terabyte arrays will only work effectively on dynamic RAID configurations. DHCP/DNS/WINS systems can be booted with the latest Windows OS without the GUI to utilize all resources for the processes on virtualized domains as well as making sure the environment is completely flexible. All these factors enable the admins to make use of such "good enough" RAID configurations due to the flexibility of the other solutions they have implemented. Only use the RAID 5 with hot spare configurations on systems that do not have ANY data backup needs, like on servers that are members of DNS round robins, while utilizing HOME drive storage solutions on dynamic terabyte arrays. The holistic (whole environment) solution approach to your data integrity assures known points of failure have dynamic contingencies.

dj4904
dj4904

The % in the first paragraph is the write penalty at the block level. While the array may consist of many disks (the stripe) the writes occur at the block level and by definition a block cannot span disks. For a single stripe in an array using a RAID5 implementation of 11 disks, there would exist 10 blocks of data on 10 disks, and 1 block of parity on 1 disk. For a single write consisting of the 1-byte char "A" to the array, 1 block on 1 of the 10 data block disks will be written to, and the 1 disk with the parity block will be written to. The same write to a RAID6 array of 12 disks would write to 1 of the 10 data block disks, and 2 separate blocks to the 2 parity blocks. I hope this makes more sense. -=dave

Florinel
Florinel

I'm sure that you know all about those "needs"! I don't know what are "capex" and "opex" also, but if I will want to know I will find some details about these.

Editor's Picks