PREFAIL: A Programmable Tool for Multiple-Failure Injection

As hardware failures are no longer rare in the era of cloud computing, cloud software systems must "Prevail" against multiple, diverse failures that are likely to occur. Testing software against multiple failures poses the problem of combinatorial explosion of multiple failures. To address this problem, the authors present PreFail, a programmable failure-injection tool that enables testers to write a wide range of policies to prune down the large space of multiple failures. They integrate PreFail to three cloud software systems (HDFS, Cassandra, and ZooKeeper), show a wide variety of useful pruning policies that they can write for them, and evaluate the speed-ups in testing time that they obtain by using the policies.