Title
An approach for oracle data selection criterion.
Abstract
Testing activities involve the execution of a program under test using input data
and the examination of the test results to determine success or failure. One of most
important tasks in examining test results is to choose the variables to observe, called
oracle data, that are used in the examination. Since it is often infeasible to examine
all variables of the program under test, we have to choose a subset of the program
variables as oracle data. If we can choose variables which can reveal more errors than
others, it can significantly increase test effectiveness. A challenge of selecting oracle
data is to predict the capability of variables to reveal errors when examined. The
work in this dissertation addresses the problem of choosing oracle data to increase
test effectiveness.
To predict the error finding capability of variables, we focus on the error propagation
behavior of a system. An error in a system can propagate to other variables
if those variables are computationally related to the error. Sometimes, however, the
error can be masked out during computation. To analyze such error propagation behavior,
we propose a novel mechanism of error propagation analysis which estimates
error propagation behavior through a static analysis of the code. The error propagation
analysis computes the error propagation probability for each variable, and this quantitative measure represents the error finding capability of variables.
The second contribution of this work is to establish an oracle data selection criterion.
In a naive approach, we may simply pick the variables with the highest error
finding capability. This, however, ignored the possibility that several variables may
reveal the same error making the selection suboptimal. Our criterion enables us to
choose oracle data to increase test effectiveness while the repeated detections of an error are minimized. The strength of our oracle data selection criterion is its ability
to choose oracle data that is effective with a small number of variables in the set of
oracle data.
To evaluate the effectiveness of our oracle data selection criterion, we developed
an error propagation analysis tool that ranks system variables based on their error
finding capabilities. We performed experiments on four sample systems to check the
error finding effectiveness of our oracle data selection criterion. The experiment shows
promising results in terms of error finding effectiveness. In addition, we investigated
the sensitivity of our oracle data selection criterion to changes in the assumptions
underlying the approach, and the results indicate that our technique is not very sensitive to even extremely skewed assumptions.
Description
University of Minnesota Ph.D. dissertation. September 2010. Major: Computer science. Advisor: Dr. Mats P.E. Heimdahl. 1 computer file (PDF); xi, 129 pages, appendices A. Ill. (some col.)
Suggested Citation
Park, Myung-Hwan.
(2010).
An approach for oracle data selection criterion..
Retrieved from the University of Minnesota Digital Conservancy,
https://hdl.handle.net/11299/99160.