Between Dec 19, 2024 and Jan 2, 2025, datasets can be submitted to DRUM but will not be processed until after the break. Staff will not be available to answer email during this period, and will not be able to provide DOIs until after Jan 2. If you are in need of a DOI during this period, consider Dryad or OpenICPSR. Submission responses to the UDC may also be delayed during this time.

Browsing by Author "George, Devaraj"

Now showing 1 - 3 of 3
  • Thumbnail Image
    Item
    On the effectiveness of specification-based structural test-coverage criteria as test-data generators for safety-critical systems
    (2012-10) George, Devaraj
    Specification based testing aims to reduce the cost of testing and increase the reliability of systems that are of a safety critical nature. A major benefit of a formal specification is the ability to automatically construct test sequences that can be executed on the implementation software. A significant portion of the testing effort in such systems requires us to demonstrate test coverage as mandated by regulatory agencies. Specification based structural test-coverage criteria that mimic source code based criteria offer us a way to analyze the structure of the specification. This typically requires coverage of certain elements such as states, branches, and decisions. These formal specifications also provide us with a means to automatically generate test sequences to satisfy test coverage. Since one of the goals of software testing is to demonstrate the existence of faults, selection of test sequences that can reveal faults is of paramount importance. Nevertheless, the relationship between test-coverage criteria and fault detection is not well established in testing literature. In this dissertation, we investigate the effectiveness of test-coverage criteria when used to drive test-data generation in the safety-critical systems domain. We provide two core contributions. First, due to the lack of sufficient evidence in testing research regarding the quality of test sets generated to satisfy test-coverage criteria, we empirically evaluate the fault-finding ability of test-sets generated to various test coverage criteria proposed in the testing literature. Second, we study the effect of test-suite reduction techniques on the generated test-data sets to empirically evaluate the sensitivity of test-coverage criteria to test-suite reduction techniques. Our findings have raised serious doubts about the use of test-coverage criteria as test-data generators in this domain. In the initial studies conducted, test sequences generated to these coverage criteria perform significantly worse at fault detection when compared to random testing that uses the same effort measured in terms of time to generate and run tests for structural test-coverage criteria such as transition coverage. In the expanded study which was conducted following the initial studies, we evaluated the fault detection effectiveness of test suites reduced to satisfy both branch and MC/DC coverage criteria against a reduced test suite of equal size using a set of random test data. The results from the expanded study validate our earlier findings and have provided us with solid statistical evidence confirming that satisfaction of a highly complex coverage criterion alone is a poor indication of test suite quality. The findings from our studies indicate a need for methods to determine test adequacy that not only provide the desired coverage, but also lend themselves as targets for automated test generation techniques. These criteria must address the problem holistically to account for all factors influencing the quality of testing, including the program structure, the nature of the state space of the system under test, the test oracle used, and finally, the test generation mechanism itself. In addition, we find that reduction techniques designed to minimize the size of a test suite while maintaining structural coverage may significantly reduce the fault-finding effectiveness of the test suite.
  • Thumbnail Image
    Item
    Specification Test Coverage Adequacy Criteria = Specification Test Generation Inadequacy Criteria?
    (2004) Heimdahl, Mats; George, Devaraj; Weber, Robert
    The successful analysis technique model checking can be employed as a test-case generation technique to generate tests from formal models. When using a model checker for test case generation, we leverage the witness (or counter-example) generation capability of model-checkers for constructing test cases. Test criteria are expressed as temporal properties and the witness traces generated for these properties are instantiated to create complete test sequences, satisfying the criteria. In this report we describe an experiment where we investigate the fault finding capability of test suites generated to provide three specification coverage metrics proposed in the literature (state , transition, and decision coverage). Our findings indicate that although the coverage may seem reasonable to measure the adequacy of a test suite, they are unsuitable when used to generate test suites. In short, the generated test sequences technically provide adequate coverage, but do so in a way that tests only a small portion of the formal model. We conclude that automated testing techniques must be pursued with great caution and that new coverage criteria targeting formal specifications are needed.
  • Thumbnail Image
    Item
    Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing
    (2004) Heimdahl, Mats; George, Devaraj
    Model checking techniques can be successfully employed as a test case generation technique to generate tests from formal models. The number of tests cases produced, however, is typically large for complex coverage criteria such as MCDC. Test-suite reduction can provide us with a smaller set of test cases that preserve the original coverage---often a dramatically smaller set. One potential drawback with test-suite reduction is that this might affect the quality of the test-suite in terms of fault finding. Previous empirical studies provide conflicting evidence on this issue. To further investigate the problem and determine its effect when testing formal models of software, we performed an experiment using a large case example of a Flight Guidance System, generated reduced test-suites for a variety of structural coverage criteria while preserving coverage, and recorded their fault finding effectiveness. Our results show that the size of the specification based test-suites can be dramatically reduced and that the fault detection of the reduced test-suites is adversely affected. In this report we describe our experiment, analyze the results, and discuss the implications for testing based on formal specifications.