Disaster Recovery

How to protect yourself from RAID-related Unrecoverable Read Errors (UREs)

Seeing an Unrecoverable Read Error (URE) during a RAID rebuild can ruin your entire day. Scott Lowe talks about UREs and how you can avoid falling victim to this silent threat.

If you are ever rebuilding a RAID system, Unrecoverable Read Error (URE) is one term you don't want to learn about the hard way. As the name implies, a URE makes for a really bad day, as it can stop a RAID rebuild in its track, essentially making the entire RAID volume unusable.

I won't go into a lot of detail about the "why" behind what causes a URE because many other very smart people have already done a good job of explaining it. (Admittedly, some of the warnings might be sensationalist, and there seems to be some confusion on terminology, but do your own math to see if you might have a serious problem.) What I will do is provide you with tips on how to make sure that you don't fall victim to these errors.

1. Don't use RAID 5 if you plan to use large nonenterprise-grade SATA disks.

When it comes to reliability, enterprise grade disks are generally at least one order of magnitude more reliable than their nonenterprise counterparts. If you believe what has been written about UREs, as the size of disks increases and as more disks are added to RAID 5 arrays, the likelihood of total data loss across the entire RAID volume begins to get into dangerous territory.

If you're trying to build a 14 disk RAID array using 1.5TB or 2TB SATA disks you bought at Best Buy, consider using RAID 6 or RAID 10 instead of RAID 5. RAID 6's dual parity mechanism provides additional cushion in the event of drive failures (at the cost of performance), while RAID 10 setups can lose up to half the disks before incurring data loss.

If you're really intent on creating a large array and want minimal RAID overhead, you could even consider RAID 50, which is a RAID 0 of RAID 5 arrays. If you were going to use RAID 50 for that 14 disk array, you'd create three four-disk RAID 5 arrays and do RAID 0 across those and use the remaining two disks as spares.

2. Don't use cheap hardware if you need to push limits.

Yes, budgets are tight, but don't risk your data. Buy enterprise-grade disks such as SAS, fibre channel disks, or at the very least high-grade SATA disks with a higher mean time between failure rating.

If you're using disks that have a bit error rate of 1 in 10^14, you're using a cheap disk. Enterprise-class disks will have a bit error rate of 1 in 10^15 or, better yet, 1 in 10^16, which makes this class of disk much less susceptible to UREs.

Note: I'm not necessarily saying that you should buy SAS disks rather than SATA disks, but do look for disks with a reasonable bit error rate.

3. If you need a lot of disks for spindle performance, use smaller capacities.

There are times when quantity outweighs capacity. For example, when your disk system is constrained by IOPS, throwing more disk spindles at the solution can fix the problem, so in some cases, you might want to have a bunch of disks in a single RAID array. If you do, use smaller capacity disks.

One of the main concerns around the possibility of encountering a URE lies in the sheer amount of time it takes to rebuild after the loss of a massive disk, such as a 1TB or 2TB disk. When disk sizes were smaller, the window of opportunity for data loss was a lot smaller since smaller disks rebuild more quickly. As disk sizes continue to grow, this window of opportunity for data loss grows, and the problem will become more serious unless manufacturers can manage to produce drives with lower bit error rates.

4. Backups, backups, backups

Someone actually asked me once why we still do backups when we use RAID on all of our servers. No matter how reliable disks get, and no matter how far away you are from even the potential of a URE, nothing replaces reliable backups. If you ever run into a URE, you'll need reliable backups.

5. Be patient

Storage needs constantly increase, and RAID 6 is simply not the answer for everyone that becomes uncomfortable with RAID 5. We have a market need and someone, somewhere will come along and fill that need with a new way to provide robust redundancy. We've also seen newer data protection methods in use in products, such as Windows Home Server and Drobo's line of devices, and it's only a matter of time before similar methods -- or completely new methods -- become available for enterprise gear.

Additional TechRepublic resources

Want to keep up with Scott Lowe's posts on TechRepublic?

About

Since 1994, Scott Lowe has been providing technology solutions to a variety of organizations. After spending 10 years in multiple CIO roles, Scott is now an independent consultant, blogger, author, owner of The 1610 Group, and a Senior IT Executive w...

13 comments
RPip
RPip

Forgive my ignorance, but this is hitting home right now with the hardware our company uses. Why does the article focus on SATA vs. SAS drives and not mention SCSI? Are we too far behind the curve? Thanks, RP

nittmann
nittmann

you cannot protect yourself from any unrecoverable error ever. However, you can minimize the impact by: - never ever use non-enterprise grade HDs for any production: reason: when HDs are manufactured, there are very tight tolerances required in the spindle and especially arm bearings. The bearings are magnetic fluid bearings. Now a manufacturer will of course not throw out >60% of production that does slightly fall out of these tight tolerances (to keep the liquid where it is, but not to grind surface to surface): Parts triage provides tolerance classes (like in car crankshafts,e.g., or main bearings), which when fit together result in well working drives. However, the liquid will not really stay where it should. Non-enterprise drives do need rest cycles to have the magnetic bearing reflow into place: 8h duty cycle is the rule of thumb. Integrating those drives into 24/7 operations is calling for disaster, especially in RAID configurations, where longer seek times due to bad bearings result in the array falling apart. This is the major reason for raid arrays to all of a sudden fall apart, needing to be restarted, or even rebuilt. This will not 'protect' but largely reduce the occurrence. And monitoring SMART data is vital: a drive may not pick up any seek errors. The moment seek error bursts occur, a drive has to be removed from a raid array (it can very well serve as a single drive in a desktop). Rebuilding high usage arrays digs off performance, a large database may become practically unusable due to the database itself killing threads due to over-long execution times. Architecture: use X-0 (50,60) configurations for online data, X (5,6) configurations for nearline and online backups. Mirrors are always presenting complete data, rebuild faster, no read penalty for checksum recalculations. Don't buy 'cheap' storage arrays from cutthroat dumpster divers who use low quality drives: your data is valuable!

zackers
zackers

You cited using smaller capacities when using multiple spindles for greater IOPS. I don't think the big problem is with UREs that occur when the rebuild is going on as you state. This is a relatively low probability even with large capacity disks that require long rebuilds. A large fraction of UREs occur when the data was originally written. Another large fraction of UREs grow silently all the time, may be around for days, weeks or months, and aren't discovered until you do an actual rebuild. The whole bit error rate concept is somewhat misleading, since it implies actual reads must be done for an URE to occur. The truth is that a disk powered down sitting on a shelf with no reads is also subject to UREs appearing. The real advantage of smaller capacity disks involved in the RAID stripes is that less data is required across all the disks to do the rebuild, and hence has a smaller chance of coming across an URE somewhere during the rebuild. On the other hand, with more spindles to get the same total capacity as with larger disks, the probability of a disk dying due to mechanical or other problems not related to UREs also goes up (not to mention your power and cooling bill). Of course, using higher quality disks also reduces this problem.

lan_ops
lan_ops

There are better solutions out there than raid 5 or Raid 6 if you do not need the performance boost of Striping. Have a look at UNraid by Lime Tech. I do not work for them but I do use their product after many years of Raid 5 and 6 Hardware and software. http://www.lime-technology.com

beedie
beedie

Don't forget the other reason for having backups - software can corrupt the data, just as well as hardware.

bzeuli
bzeuli

One other consideration is a combination of Raid 6 and a collection of Raid 10 (or 1). I am in the document management industry and for every 1,000 byte record we save in the database (Raid 6) we also save a linked document into the file system (Raid 1) that might be many megabytes. And while we need one large volume for the database, we spread the actual documents across many individual Raid 1 volumes. We believe this increases reliabilty and saves power as the Raid 1 volumes automatically spin down and only spin up as accessed.

knowlengr
knowlengr

Well put. Been burned w/ RAID 5. There's a lot of pain, suffering and expense finding and implementing alternatives, though.

steve-f
steve-f

Nice article. Another reason to backup as well as using RAID: RAID doesn't save you from accidental deletion or corruption of data.

speculatrix
speculatrix

IMHO, drives tend to fail when they are powered up, it's a combination of thermal stress from the temperature change and the stress on the motor. You need to balance this with extending the life of a drive by leaving it turned off. My gut feel is that if the drive is used more than an hour or two a day, keeping it running in a low-power idling state is better than switching it off.

klapper
klapper

I once had a SCSI backplane slowly die and the problem wasn't reported. First it appeared that one drive had died. We replaced it, but then we began to notice OS corruptions. Next thing we knew, the entire RAID5 array was dead. Thank goodness for the backup.

Editor's Picks