An approach for oracle data selection criterion.

Abstract

Testing activities involve the execution of a program under test using input data and the examination of the test results to determine success or failure. One of most important tasks in examining test results is to choose the variables to observe, called oracle data, that are used in the examination. Since it is often infeasible to examine all variables of the program under test, we have to choose a subset of the program variables as oracle data. If we can choose variables which can reveal more errors than others, it can significantly increase test effectiveness. A challenge of selecting oracle data is to predict the capability of variables to reveal errors when examined. The work in this dissertation addresses the problem of choosing oracle data to increase test effectiveness. To predict the error finding capability of variables, we focus on the error propagation behavior of a system. An error in a system can propagate to other variables if those variables are computationally related to the error. Sometimes, however, the error can be masked out during computation. To analyze such error propagation behavior, we propose a novel mechanism of error propagation analysis which estimates error propagation behavior through a static analysis of the code. The error propagation analysis computes the error propagation probability for each variable, and this quantitative measure represents the error finding capability of variables. The second contribution of this work is to establish an oracle data selection criterion. In a naive approach, we may simply pick the variables with the highest error finding capability. This, however, ignored the possibility that several variables may reveal the same error making the selection suboptimal. Our criterion enables us to choose oracle data to increase test effectiveness while the repeated detections of an error are minimized. The strength of our oracle data selection criterion is its ability to choose oracle data that is effective with a small number of variables in the set of oracle data. To evaluate the effectiveness of our oracle data selection criterion, we developed an error propagation analysis tool that ranks system variables based on their error finding capabilities. We performed experiments on four sample systems to check the error finding effectiveness of our oracle data selection criterion. The experiment shows promising results in terms of error finding effectiveness. In addition, we investigated the sensitivity of our oracle data selection criterion to changes in the assumptions underlying the approach, and the results indicate that our technique is not very sensitive to even extremely skewed assumptions.