Challenges in Testing Next Generation CPS Systems
2017
View/Download File
Persistent link to this item
Statistics
View StatisticsJournal Title
Journal ISSN
Volume Title
Title
Challenges in Testing Next Generation CPS Systems
Authors
Published Date
2017
Publisher
Type
Report
Abstract
Testing cyber-physical systems presents a unique set of testing challenges: heterogeneity, timing, and, especially, observability. In particular, some of the mechanisms that are designed to make embedded software robust are the same mechanisms that present challenges for automated testing techniques: e.g., rate limiting, fault masking, and debounce logic, which can lead to long lags between problematic inputs and their manifestation in system outputs. In addition, much of the control behavior of CPS is mathematically intensive more than "branchy". For such systems, traditional coverage metrics, which focus on reaching program statements or mutants (as in weak mutation), rather than observing their affect on system outputs, are likely to lead to poor fault-finding in practice. I will describe some recent work in the area of test generation and code metrics that begins to address these issues.
I then argue that this mismatch will become significantly worse for when examining CPS systems that incorporate machine learning: the complexity of the non-linear mathematics used defeats all known symbolic techniques and the lack of a traditional branching structure renders traditional code coverage metrics fairly meaningless when using search-based testing. In fact, a recent paper abstract claims: "We propose a novel control flow obfuscation design based on the incomprehensibility of artificial neural networks to fight against reverse engineering tools including concolic testing." Traditional testing of such systems has focused on checking their behavior against a large corpus of examples; unfortunately, it is known that such systems are often not robust against malicious inputs even after extensive training and testing. I hope to challenge participants to focus research into such mathematically intensive systems and to begin to create rigorous metrics that are applicable to such systems.
Keywords
Description
Associated research group: Critical Systems Research Group
Related to
Replaces
License
Series/Report Number
Funding information
Isbn identifier
Doi identifier
Previously Published Citation
Testing Embedded and Cyber Physical Systems Workshop
Suggested citation
Content distributed via the University Digital Conservancy may be subject to additional license and use restrictions applied by the depositor. By using these files, users agree to the Terms of Use. Materials in the UDC may contain content that is disturbing and/or harmful. For more information, please see our statement on harmful content in digital repositories.