Do software engineering practitioners cite software testing research in their online articles? A larger scale replication

Background: Software engineering (SE) research continues to study the degree to which practitioners perceive research as relevant to practice. Such studies typically comprise surveys of practitioner opinions. In a preliminary, and relatively small scale, study of online articles we previously found few explicit citations to software testing research. Our previous study provided an in situ complement to the typical survey study, however the findings of the previous study were limited by the size of our sample. Objective: To further investigate whether and how practitioners cite software testing research in the grey literature, by using a larger and more diverse dataset. Method: We analyse four distinct datasets totalling over 400,000 online articles with approx. 2M external citations. Two datasets were generated by crawling predefined domains and two were generated by applying heuristics, developed in prior research, in Google searches. Citations are classified and then analysed. Results: We find a (very) low percentage of citations to research. Conclusion: Our replication corroborates our preliminary study and findings from others. In relative terms, topic–specific searches appear to return results that contain articles with more citations. Our results and method provide a basis for benchmarking.