Investigating the Connection between Behavioural Diversity and Mutation Operators in Software Testing

Abstract

[Context and Motivation] Mutation testing supports testers by assessing the quality of a test suite by automatically introducing systematically generated faults in the software and testing if the test cases fail the mutated source code. Despite offering insights on test behaviour, the costs associated with mutation testing are very high, since mutation tools require multiple test runs on various altered versions of the source code. Recently, researchers proposed the usage of mutation testing to capture the behavioural diversity of test cases, particularly, to improve effectiveness in test prioritisation. [Question/problem] However, the correlation between behavioural diversity and mutation operators are still unknown. For instance, investigating whether mutants generated by different mutation operators are killed by tests with similar behaviour. Mutation testing is very costly and behavioural diversity can reveal relationships to make it more efficient. [Results] Our thesis, reports on an experiment, in which we use mutation tools on seven open-source projects to generate and analyse the relationships between behavioural diversity and those mutation operators. Our results reveal that the test behaviour of some mutation operators are highly correlated to each other (e.g., arithmetical and relational operators), whereas some are not (conditional and arithmetical operators). Moreover, we investigate those correlations in connection to subsumed mutants and how test cases clusters together based on behavioural diversity for different mutation operators. [Contribution] The correlations revealed in our findings bring awareness to relationships between test diversity and mutation testing which paves the path for researchers and practitioners to look more deeply at how to make the mutation tools work more effectively and efficiently.