Browsing by Author "You, Dongjiang"

Now showing 1 - 9 of 9
  • Thumbnail Image
    Item
    A Simulation Study on Some Search Algorithms for Regression Test Case Prioritization
    (IEEE, 2010) You, Dongjiang
    Test case prioritization is an approach aiming at increasing the rate of faults detection during the testing phase, by reordering test case execution. Many techniques for regression test case prioritization have been proposed. In this paper, we perform a simulation experiment to study five search algorithms for test case prioritization and compare the performance of these algorithms. The target of the study is to have an in-depth investigation and improve the generality of the comparison results. The simulation study provides two useful guidelines: (1) Two search algorithms, Additional Greedy Algorithm and 2-Optimal Greedy Algorithm, outperform the other three search algorithms in most cases. (2) The performance of the five search algorithms will be affected by the overlap of test cases with regard to test requirements.
  • Thumbnail Image
    Item
    An Empirical Study on the Effectiveness of Time-Aware Test Case Prioritization Techniques
    (ACM, 2011) You, Dongjiang
    Regression testing is often performed with a time budget and it does not allow executing all test cases. Test case prioritization techniques re-order test cases to increase the rate of fault detection. Several time-aware test case prioritization techniques have been proposed to satisfy a time budget. Since it is difficult to collect the time cost of each test case in some cases, a natural question is whether it is worth using such information when prioritizing test cases. In this paper, two most popular criteria: statement coverage and fault detection are considered for time-aware test case prioritization. We investigate whether the time cost of each test case affects the effectiveness of prioritization techniques, i.e. the rate of statement coverage and the rate of fault detection. Our empirical study shows that: although the techniques considering the time cost of each test case are slightly better than the techniques not considering such information in some cases, they have no significant difference in most cases.
  • Thumbnail Image
    Item
    Are We There Yet? Determining the Adequacy of Formalized Requirements and Test Suites
    (Springer, 2015) Murugesan, Anitha; Whalen, Michael; Rungta, Neha; Tkachuk, Oksana; Person, Suzette; Heimdahl, Mats; You, Dongjiang
    Structural coverage metrics have traditionally categorized code as either covered or uncovered. Recent work presents a stronger notion of coverage, checked coverage, which counts only statements whose execution contributes to an outcome checked by an oracle. While this notion of coverage addresses the adequacy of the oracle, for Model-Based Development of safety critical systems, it is still not enough; we are also interested in how much of the oracle is covered, and whether the values of program variables are masked when the oracle is evaluated. Such information can help system engineers identify missing requirements as well as missing test cases. In this work, we combine results from checked coverage with results from requirements coverage to help provide insight to engineers as to whether the requirements or the test suite need to be improved. We implement a dynamic backward slicing technique and evaluate it on several systems developed in Simulink. The results of our preliminary study show that even for systems with comprehensive test suites and good sets of requirements, our approach can identify cases where more tests or more requirements are needed to improve coverage numbers.
  • Thumbnail Image
    Item
    Automated Model-based Test Generation for Platform-specific Implementations
    (2016-06) You, Dongjiang
    In model-based testing of safety-critical systems, structural coverage criteria have been widely used to measure test suite adequacy as well as a target when generating tests. We have found that the fault-finding effectiveness of tests satisfying structural coverage criteria is highly dependent on program structure; and even if the faulty code is exercised, its effect may not be observable at the output. To address these problems, we define observability as a desirable attribute of testing to mandate that the effect of exercising a structural part must manifest itself at a subsequent observable point in the program. We further propose an incremental test generation approach that combines the notion of observability and dynamic symbolic execution. Our results show that the notion of observability together with the incremental test generation approach are effective at detecting faults, robust to program restructuring, and efficient in generating tests. On the other hand, advances in automated test generation from system models do not always translate to realizable benefits in terms of testing an implementation of the system, because platform-specific details are often abstracted away to make the models amenable to various analyses. Testing an implementation to expose non-conformance to such a model requires reconciling differences arising from these abstractions. Previously proposed approaches address this by being reactively permissive: passing criteria are relaxed to reduce false positives, but may increase false negatives, which is particularly bothersome for safety-critical systems. To address this concern, we propose an automated approach that is proactively adaptive: test stimuli and system responses are suitably modified taking into account platform-specific aspects so that the modified test -- when executed on the platform-specific implementation -- exercises the intended scenario captured in the original model-based test. We show that our framework eliminates false negatives while keeping the number of false positives low for a variety of platform-specific implementations.
  • Thumbnail Image
    Item
    Efficient Observability-based Test Generation by Dynamic Symbolic Execution
    (IEEE, 2015) You, Dongjiang; Rayadurgam, Sanjai; Whalen, Michael; Heimdahl, Mats; Gay, Gregory
    Structural coverage metrics have been widely used to measure test suite adequacy as well as to generate test cases. In previous investigations, we have found that the fault-finding effectiveness of tests satisfying structural coverage criteria is highly dependent on program syntax – even if the faulty code is exercised, its effect may not be observable at the output. To address these problems, observability-based coverage metrics have been defined. Specifically, Observable MC/DC (OMC/DC) is a criterion that appears to be both more effective at detecting faults and more robust to program restructuring than MC/DC. Traditional counterexample-based test generation for OMC/DC, however, can be infeasible on large systems. In this study, we propose an incremental test generation approach that combines the notion of observability with dynamic symbolic execution. We evaluated the efficiency and effectiveness of our approach using seven systems from the avionics and medical device domains. Our results show that the incremental approach requires much lower generation time, while achieving even higher fault finding effectiveness compared with regular OMC/DC generation.
  • Thumbnail Image
    Item
    Executing Model-based Tests on Platform-specific Implementations
    (IEEE, 2015) You, Dongjiang; Rayadurgam, Sanjai; Heimdahl, Mats; Komp, John; Kim, Baek-Gyu; Sokolsky, Oleg
    Model-based testing of embedded real-time systems is challenging because platform-specific details are often abstracted away to make the models amenable to various analyses. Testing an implementation to expose non-conformance to such a model requires reconciling differences arising from these abstractions. Due to stateful behavior, naive comparisons of model and system behaviors often fail causing numerous false positives. Previously proposed approaches address this by being reactively permissive: passing criteria are relaxed to reduce false positives, but may increase false negatives, which is particularly bothersome for safety-critical systems. To address this concern, we propose an automated approach that is proactively adaptive: test stimuli and system responses are suitably modified taking into account platform-specific aspects so that the modified test when executed on the platform-specific implementation exercises the intended scenario captured in the original model-based test. We show that the new framework eliminates false negatives while keeping the number of false positives low for a variety of platform-specific configurations.
  • Thumbnail Image
    Item
    Mutation Selection: Some Could be Better than All
    (2011) You, Dongjiang
    In previous research, many mutation selection techniques have been proposed to reduce the cost of mutation analysis. After a mutant subset is selected, researchers could obtain a test suite which can detect all mutants in the mutant subset. Then they run all mutants over this test suite, and the detection ratio to all mutants is used to evaluate the effectiveness of mutation selection techniques. The higher the ratio is, the better this selection technique is. Obviously, this measurement has a presumption that the set of all mutants is the best to evaluate test cases. However, there is no clearly evidence to support this presumption. So we conducted an experiment to answer the question whether the set of all mutants is the best to evaluate test cases. In this paper, our experiment results show that a subset of mutants may be more similar to faults than all the mutants. Two evaluation metrics were used to measure the similarity – rank and distance. This finding reveals that it may be more appropriate to use a subset rather than all the mutants at hand to evaluate the fault detection capability of test cases.
  • Thumbnail Image
    Item
    Observable Modified Condition/Decision Coverage
    (IEEE, 2013) Whalen, Michael; Gay, Gregory; You, Dongjiang; Heimdahl, Mats; Staats, Matt
    In many critical systems domains, test suite adequacy is currently measured using structural coverage metrics over the source code. Of particular interest is the modified condition/decision coverage (MC/DC) criterion required for, e.g., critical avionics systems. In previous investigations we have found that the efficacy of such test suites is highly dependent on the structure of the program under test and the choice of variables monitored by the oracle. MC/DC adequate tests would frequently exercise faulty code, but the effects of the faults would not propagate to the monitored oracle variables. In this report, we combine the MC/DC coverage metric with a notion of observability that helps ensure that the result of a fault encountered when covering a structural obligation propagates to a monitored variable; we term this new coverage criterion Observable MC/DC (OMC/DC). We hypothesize this path requirement will make structural coverage metrics 1.) more effective at revealing faults, 2.) more robust to changes in program structure, and 3.) more robust to the choice of variables monitored. We assess the efficacy and sensitivity to program structure of OMC/DC as compared to masking MC/DC using four subsystems from the civil avionics domain and the control logic of a microwave. We have found that test suites satisfying OMC/DC are significantly more effective than test suites satisfying MC/DC, revealing up to 88% more faults, and are less sensitive to program structure and the choice of monitored variables.
  • Thumbnail Image
    Item
    Practical Aspects of Building a Constrained Random Test Framework for Safety-critical Embedded Systems
    (ACM, 2014) You, Dongjiang; Amundson, Isaac; Hareland, Scott; Rayadurgam, Sanjai
    In the safety-critical embedded system industry, one of the key challenges is to demonstrate the robustness and dependability of the product prior to market release, which is typically done using various verification and validation (V&V) strategies. Directed verification testing is a common strategy that performs black-box testing at the system level; however, it only samples a small set of specific system behaviors and requires heavily manual effort. In this paper, we describe our experience and lessons learned of applying the concept of constrained random testing on safety-critical embedded systems as a complimentary testing methodology. Constrained random testing enables us to cover many more system behaviors through random input variations, random fault injections, and automatic output comparisons. Additionally, it can reduce manual effort and increase confidence on the dependability of both firmware and hardware.