Preprint
A PDF version of the manuscript accepted for presentation at the 39th International Conference on Software Engineering (ICSE'17) is available here.
Abstract
We introduce covrel, an adaptive software testing approach based on the combined use of operational profile and coverage spectrum, with the ultimate goal of improving the delivered reliability of the program under test. Operational profile-based testing is a black-box technique that selects the test cases having the largest impact on failure probability in operation; as such, it is considered well suited when reliability is a major concern. Program spectrum is a characterization of a program’s behavior in terms of the code entities (e.g., branches, statements, functions) that are covered as the program executes. The driving idea of covrel is to complement operational profile information with white-box coverage measures based on count spectra, so as to dynamically select the most effective test cases for reliability improvement. In particular we bias operational profile-based test selection towards those entities that have been less frequently covered. We assess the approach through experiments with a number of subjects commonly used in software testing research, and compare it with traditional operational and coverage testing. The results show that exploiting operational and coverage data in a combined adaptive way actually pays in terms of reliability improvement.
Experiment Results
- How does covrel compare with operational profile based testing in terms of delivered reliability evolution? Here we provide detailed results of our experiments comparing covrel with operational profile based testing. These are packaged in a number of files in CSV format: each experiment includes the six configurations described in Section IV of our paper (three coverage criteria and two types of fault matrix), and produces five CSV files with many more details than those presented in the paper, plus a textual file with the console output, which allows post-execution reconstruction of the progress of the experiment. Go to the results.
- How does covrel compare with traditional white-box coverage-based selection heuristics? Additional results are also available for our comparison between covrel and traditional white-box coverage-based selection heuristics. Go to the results.