Abstract
We present an adaptive software testing strategy for test case allocation, selection and generation, based on the combined use of operational profile and coverage spectrum, aimed at achieving high delivered reliability of the program under test. Operational profile-based testing is a black-box technique considered well suited when reliability is a major concern, as it selects the test cases having the largest impact on failure probability in operation. Coverage spectrum is a characterization of a program’s behavior in terms of the code entities (e.g., branches, statements, functions) that are covered as the program executes. The proposed strategy - named covrel+ - complements operational profile information with white-box coverage measures, so as to adaptively select/generate the most effective test cases for improving reliability as testing proceeds. We assess covrel+ through experiments with subjects commonly used in software testing research, comparing results with traditional operational testing. The results show that exploiting operational and coverage data in an integrated adaptive way allows generally to outperform operational testing at achieving a given reliability target, or at detecting faults under the same testing budget, and that covrel+ has greater ability than operational testing in detecting hard-to-detect faults.
Verifiability
Here we provide multiple artifacts to support independent verification of our experiments, as well as repeatability with other subject programs. You can get a prototypal version of covrel application with several modules written in Java and Python here. All the subjects investigated in our experiments can be taken directly from the SIR repository. Alternatively, below we provide, for each subject, a .zip file containing all the material required for running the experiments reported in our paper.- Grep v1
- Grep v2
- Grep v3
- Grep v4
- Gzip v1
- Gzip v2
- Gzip v4
- Gzip v5
- Sed v2
- Sed v3
- Sed v4
- Sed v5
- Sed v6
- Sed v7
- Flex v1
- Flex v2
- Flex v3
- Flex v4