Comparing automated visual GUI testing tools: An industrial case study

Vahid Garousi, İhsan Berk Işık, Ahmet Zeki Boyraz, Wasif Afzal, Berker Baydan, Burak Yolaçan, Adem Çağlar, Seçkin Çaylak, Kadir Herkiloğlu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Citations (Scopus)
105 Downloads (Pure)

Abstract

Visual GUI testing (VGT) is a tool-driven technique, which uses image recognition for interaction and assertion of the behaviour of system under test. Motivated by a real industrial need, in the context of a large Turkish software and systems company providing solutions in the areas of defense and IT sector, we systematically planned and applied a VGT project in this industrial context. The goal of the initial phase of the project was to empirically evaluate two well-known VGT tools (Sikuli and JAutomate) to help the company select the best tool for a given testing project. Our results show that both two tools suffer from similar test ‘Replay’ problems such as the inability to find smaller-sized images. The repeatability of test executions was better for JAutomate in case of one of the two software under test (SUT) while it was comparable for the other. In terms of test development effort, for both tools, there were high correlations with number of steps in test suites, however the effort is reduced if test code is reused. The study has already provided benefits to the test engineers and managers in the company by increasing the know-how in the company w.r.t. VGT, and by identifying the challenges and their workarounds in using the tools. The industrial case study in this paper intends to add to the body of evidence in VGT and help other researchers and practitioners.

Original languageEnglish
Title of host publicationProceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing, Co-located with FSE 2017
EditorsSigrid Eldh, Wishnu Prasetya, Tanja E.J. Vos
PublisherAssociation for Computing Machinery, Inc
Pages21-28
Number of pages8
ISBN (Print)9781450351553
DOIs
Publication statusPublished - 04 Sep 2017
Externally publishedYes
Event8th ACM SIGSOFT International Workshop on Automated Software Testing, A-TEST 2017, co-located with the International ESEC/FSE Conference, FSE 2017 - Paderborn, Germany
Duration: 04 Sep 201705 Sep 2017

Conference

Conference8th ACM SIGSOFT International Workshop on Automated Software Testing, A-TEST 2017, co-located with the International ESEC/FSE Conference, FSE 2017
Country/TerritoryGermany
CityPaderborn
Period04/09/201705/09/2017

Keywords

  • Automated Testing Tools
  • Empirical Evaluation
  • Graphical User Interface (GUI) Testing
  • Industrial Case Study
  • Visual GUI Testing

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'Comparing automated visual GUI testing tools: An industrial case study'. Together they form a unique fingerprint.

Cite this