Using Runtime Testing to Detect Defects in Applications Without Test Oracles

The authors address the testing of complex, highly-configurable systems - particularly those without test oracles - by testing in the field using built-in oracles from functions' metamorphic properties. For large, complex software systems, it is typically impossible in terms of time and cost to reliably test all configuration options and all possible system states before releasing the product into the field. Even given infinite time and re-sources to test an application and all its configurations, once a product is released, the other software packages on which it depends (libraries, virtual machines, etc.) may also be up-dated; therefore, it would be impossible to test these prior to release, because they didn't exist yet.