Automatic Test Suite Generation for Scientific MATLAB Code

Abstract

Software testing is the process of finding code faults by applying tests and comparing results from the code to an oracle. Mutation testing is one of many testing techniques. A mutation is a single syntactic change to the original code. A mutation score is the percentage of mutants detected by any given test suite. So it is possible to compare the effectiveness of different test suites. Testing techniques cannot be easily applied to scientific code for two reasons. First, an oracle is usually unavailable. Second, scientific code output typically deals with real numbers rather than whole numbers. Correctness of the code depends on the tolerance level that is acceptable. Mutation sensitivity testing tackles the tolerance problem by systematically exploring what happens across a range of relative error between a mutation and the original program under test. This thesis is an extension to earlier work on mutation sensitivity testing of scientific MATLAB code. An automatic test case generation technique is proposed based on the use of a genetic algorithm. This approach allows for the creation of test suites which detect mutants at the highest possible levels of relative error. Test suites have been automatically generated for the 8 scientific functions used in earlier work and comparisons drawn with the results from existing manual test suites. As a final step, the 8 scientific functions were unit tested by using independent technologies to calculate expected outputs from the generated test inputs.