Test Cost-Effectiveness and Defect Density: A Case Study on the Android Platform

The Android operating system is one of the most popular open-source platforms in the mobile operating system market. It had a worldwide smart-phone market share of 68% at the second quarter of 2012. However, there has been little research on test coverage and test cost-effectiveness in this platform. The goal of this case study reported in this paper is to assess test coverage, fault detection effectiveness, test cost-effectiveness, and defect density in code-base of version 2.1 of the Android platform. We raise and address five research questions (RQs) in this study. Among our results are: (1) in contrary to what one would expect, for packages with larger coverage values (meaning more rigorous testing), it is not necessarily true that less defects have been reported by the users after release. Also, it is not necessarily true that components with low coverage have more defects; (2) we re-confirm (replicate) the existence of correlation between code coverage and mutation score, similar to existing studies; and (3) the package with the highest defect density (DD) in the Android code-base is Music (DD=0.19 per 1 KLOC) and the package with the lowest DD value is ContactsProvider (DD=0.0003). Results of our study will help us and other researchers to get a better view on test coverage, fault detection effectiveness, test cost-effectiveness, and defect density in Android code-base.