When and what to automate in software testing? A multi-vocal literature review

Vahid Garousi*, Mika V. Mäntylä

*Corresponding author for this work

Research output: Contribution to journalReview article

59 Citations (Scopus)
383 Downloads (Pure)


Context Many organizations see software test automation as a solution to decrease testing costs and to reduce cycle time in software development. However, establishment of automated testing may fail if test automation is not applied in the right time, right context and with the appropriate approach. Objective The decisions on when and what to automate is important since wrong decisions can lead to disappointments and major wrong expenditures (resources and efforts). To support decision making on when and what to automate, researchers and practitioners have proposed various guidelines, heuristics and factors since the early days of test automation technologies. As the number of such sources has increased, it is important to systematically categorize the current state-of-The-art and -practice, and to provide a synthesized overview. Method To achieve the above objective, we have performed a Multivocal Literature Review (MLR) study on when and what to automate in software testing. A MLR is a form of a Systematic Literature Review (SLR) which includes the grey literature (e.g., blog posts and white papers) in addition to the published (formal) literature (e.g., journal and conference papers). We searched the academic literature using the Google Scholar and the grey literature using the regular Google search engine. Results Our MLR and its results are based on 78 sources, 52 of which were grey literature and 26 were formally published sources. We used the qualitative analysis (coding) to classify the factors affecting the when- And what-to-automate questions to five groups: (1) Software Under Test (SUT)-related factors, (2) test-related factors, (3) test-tool-related factors, (4) human and organizational factors, and (5) cross-cutting and other factors. The most frequent individual factors were: need for regression testing (44 sources), economic factors (43), and maturity of SUT (39). Conclusion We show that current decision-support in software test automation provides reasonable advice for industry, and as a practical outcome of this research we have summarized it as a checklist that can be used by practitioners. However, we recommend developing systematic empirically-validated decision-support approaches as the existing advice is often unsystematic and based on weak empirical evidence.

Original languageEnglish
Pages (from-to)92-117
Number of pages26
JournalInformation and Software Technology
Publication statusPublished - 01 Aug 2016
Externally publishedYes


  • Decision support
  • Multivocal literature review
  • Software test automation
  • Systematic literature review
  • Systematic Mapping study
  • What to automate
  • When to automate

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Computer Science Applications

Fingerprint Dive into the research topics of 'When and what to automate in software testing? A multi-vocal literature review'. Together they form a unique fingerprint.

  • Cite this