Choosing the right test automation tool: A grey literature review of practitioner sources

Päivi Raulamo-Jurvanen, Mika Mäntylä, Vahid Garousi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)
58 Downloads (Pure)

Abstract

Background: Choosing the right software test automation tool is not trivial, and recent industrial surveys indicate lack of right tools as the main obstacle to test automation. Aim: In this paper, we study how practitioners tackle the problem of choosing the right test automation tool. Method: We synthesize the "voice" of the practitioners with a grey literature review originating from 53 different companies. The industry experts behind the sources had roles such as "Software Test Automation Architect", and "Principal Software Engineer". Results: Common consensus about the important criteria exists but those are not applied systematically. We summarize the scattered steps from individual sources by presenting a comprehensive process for tool evaluation with 12 steps and a total of 14 different criteria for choosing the right tool. Conclusions: The practitioners tend to have general interest in and be influenced by related grey literature as about 78% of our sources had at least 20 backlinks (a reference comparable to a citation) while the variation was between 3 and 759 backlinks. There is a plethora of different software testing tools available, yet the practitioners seem to prefer and adopt the widely known and used tools. The study helps to identify the potential pitfalls of existing processes and opportunities for comprehensive tool evaluation.

Original languageEnglish
Title of host publicationProceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE 2017
PublisherAssociation for Computing Machinery
Pages21-30
VolumePart F128635
ISBN (Electronic)9781450348041
DOIs
Publication statusPublished - 15 Jun 2017
Externally publishedYes
Event21st International Conference on Evaluation and Assessment in Software Engineering, EASE 2017 - Karlskrona, Sweden
Duration: 15 Jun 201716 Jun 2017

Conference

Conference21st International Conference on Evaluation and Assessment in Software Engineering, EASE 2017
CountrySweden
CityKarlskrona
Period15/06/201716/06/2017

Keywords

  • Grey literature review
  • Software test automation
  • Test automation tool
  • Tool selection

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software

Fingerprint Dive into the research topics of 'Choosing the right test automation tool: A grey literature review of practitioner sources'. Together they form a unique fingerprint.

  • Cite this

    Raulamo-Jurvanen, P., Mäntylä, M., & Garousi, V. (2017). Choosing the right test automation tool: A grey literature review of practitioner sources. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE 2017 (Vol. Part F128635, pp. 21-30). Association for Computing Machinery. https://doi.org/10.1145/3084226.3084252