Mining and comparing software testing metrics and evaluating the use of different test quality metric tools

Abstract

Software test metrics are often compared with one another to find the most accurate predictor of test quality, however little is written about the connections between types of metrics. Moreover, open-source software is more commonly used to extract these test metrics, but the issues researchers face when attempting to use them is under-reported. [Objective] This paper aims to investigate the correlation between coverage, mutation and diversity-based test metrics, as well as to describe the challenges involved in automating the use of test quality metric extraction tools. [Method] We conducted an MSR study on 22 open-source software repositories from Github which were mined in order to extract test quality metrics. To do this we developed a tool which combined several metric extraction tools to provide code coverage, mutation coverage and average test suite diversity values. [Results] The challenges we faced when conducting this MSR study stemmed mostly from trying to automate the test quality metric extraction process, such as finding repositories that were compatible with all of our metric extraction tools, Maven plugins interfering, costly execution time and tools not working out of the box.We then applied Spearman’s Rho to calculate the correlation between the metrics. [Conclusion] Our results show that code coverage, mutation coverage and test suite diversity all have a positive correlation with each other. We also found that automating the extraction of test metrics limited the availability of repositories due to stricter requirements.