We present an adaptive software testing strategy for test case allocation, selection and generation, based on the combined use of operational profile and coverage spectrum, aimed at achieving high delivered reliability of the program under test. Operational profile-based testing is a black-box technique considered well suited when reliability is a major concern, as it selects the test cases having the largest impact on failure probability in operation. Coverage spectrum is a characterization of a program’s behavior in terms of the code entities (e.g., branches, statements, functions) that are covered as the program executes. The proposed strategy - named covrel+ - complements operational profile information with white-box coverage measures, so as to adaptively select/generate the most effective test cases for improving reliability as testing proceeds. We assess covrel+ through experiments with subjects commonly used in software testing research, comparing results with traditional operational testing. The results show that exploiting operational and coverage data in an integrated adaptive way allows generally to outperform operational testing at achieving a given reliability target, or at detecting faults under the same testing budget, and that covrel+ has greater ability than operational testing in detecting hard-to-detect faults.
Here we provide multiple artifacts to support independent verification of our experiments, as well as repeatability with other subject programs. You can get a prototypal version of covrel application with several modules written in Java and Python here. All the subjects investigated in our experiments can be taken directly from the SIR repository. Alternatively, below we provide, for each subject, a .zip file containing all the material required for running the experiments reported in our paper.
This work builds on and expands significantly a previous version presented at the IEEE/ACM 39th International Conference on Software Engineering (ICSE) in 2017, entitled “Adaptive Coverage and Operational Profile-based Testing for Reliability Improvement”. A PDF version of the previously published work is available here.