Coverage Metrics for Requirements-Based Testing: Evaluation of Effectiveness
Loading...
View/Download File
Persistent link to this item
Statistics
View StatisticsJournal Title
Journal ISSN
Volume Title
Title
Coverage Metrics for Requirements-Based Testing: Evaluation of Effectiveness
Published Date
2010
Publisher
NASA
Type
Report
Abstract
In black-box testing, the tester creates a set of tests to exercise a system under test without regard
to the internal structure of the system. Generally, no objective metric is used to measure the
adequacy of black-box tests. In recent work, we have proposed three requirements coverage metrics,
allowing testers to objectively measure the adequacy of a black-box test suite with respect to a set
of requirements formalized as Linear Temporal Logic (LTL) properties. In this report, we evaluate
the effectiveness of these coverage metrics with respect to fault finding. Specifically, we conduct an
empirical study to investigate two questions: (1) do test suites satisfying a requirements coverage
metric provide better fault finding than randomly generated test suites of approximately the same
size?, and (2) do test suites satisfying a more rigorous requirements coverage metric provide better
fault finding than test suites satisfying a less rigorous requirements coverage metric?
Our results indicate (1) that test suites satisfying more rigorous coverage metrics provide better
fault finding than test suites satisfying less rigorous coverage metrics and (2) only one coverage
metric proposed—Unique First Cause (UFC) coverage—is sufficiently rigorous to ensure test suites
satisfying the metric outperform randomly generated test suites of similar size.
Keywords
Description
Associated research group: Critical Systems Research Group
Related to
Replaces
License
Series/Report Number
Funding information
Isbn identifier
Doi identifier
Previously Published Citation
Proceedings of the Second NASA Formal Methods Symposium
Suggested citation
Staats, Matt; Whalen, Michael; Rajan, Ajitha; Heimdahl, Mats. (2010). Coverage Metrics for Requirements-Based Testing: Evaluation of Effectiveness. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/217415.
Content distributed via the University Digital Conservancy may be subject to additional license and use restrictions applied by the depositor. By using these files, users agree to the Terms of Use. Materials in the UDC may contain content that is disturbing and/or harmful. For more information, please see our statement on harmful content in digital repositories.