The choice of test oracleâ€”the artifact that determines whether an application under test executes correctlyâ€”can significantly impact the effectiveness of the testing process. However, despite the prevalence of tools that support test input selection, little work exists for supporting oracle creation. We propose a method of supporting test oracle creation that automatically selects the oracle dataâ€”the set of variables monitored during testingâ€”for expected value test oracles. This approach is based on the use of mutation analysis to rank variables in terms of fault-finding effectiveness, thus automating the selection of the oracle data. Experimental results obtained by employing our method over six industrial systems (while varying test input types and the number of generated mutants) indicate that our methodâ€”when paired with test inputs generated either at random or to satisfy specific structural coverage criteriaâ€”may be a cost-effective approach for producing small, effective oracle data sets, with fault finding improvements over current industrial best practice of up to 1,435% observed (with typical improvements of up to 50%).
Appeared in IEEE Transactions on Software Engineering, Volume 41 (11), November, 2015
Associated research group: Critical Systems Research Group
Gay, Gregory; Staats, Matt; Whalen, Michael; Heimdahl, Mats.
Automated Oracle Data Selection Support.
Retrieved from the University of Minnesota Digital Conservancy,
Content distributed via the University of Minnesota's Digital Conservancy may be subject to additional license and use restrictions applied by the depositor.