Mauro Pezzè
Paolo Rondena
ABSTRACT
Testing software applications interacting with their graphical user interface, in short GUI testing, is both important, since it can reveal subtle and annoying bugs, and expensive, due to myriads of possible GUI interactions. Recent attempts to automate GUI testing have produced several techniques that address the problem from different perspectives, sometimes focusing only on some specific platforms, such as Android or Web, and sometimes targeting only some aspects of GUI testing, like test case generation or execution. Although GUI test case generation techniques for desktop applications were the first to be investigated, this area is still actively researched and its state of the art is continuously expanding.
In this paper we comparatively evaluate the state-of-the-art for automatic GUI test cases generation for desktop applications, by presenting a set of experimental results obtained with the main GUI testing tools for desktop applications available. The paper overviews the state of the art in GUI testing, discusses differences, similarities and complementarities among the different techniques, experimentally compares strengths and weaknesses, and pinpoints the open problems that deserve further investigation.
- Stephan Arlt, Andreas Podelski, Clement Bertolini, Martin Schaf, Indradip Banerjee, and Atif M Memon. 2012. Lightweight static analysis for GUI testing. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE '12). IEEE Computer Society, 301--310. Google Scholar
Digital Library
- Ishan Banerjee, Bao Nguyen, Vahid Garousi, and Atif Memon. 2013. Graphical user interface (GUI) testing: Systematic mapping and repository. Information and Software Technology 55, 10 (2013), 1679--1694. Google ScholarDigital Library
- Sebastian Bauersfeld and Tanja EJ Vos. 2014. User interface level testing with TESTAR; what about more sophisticated action specification and selection?. In Seminar Series on Advanced Techniques and Tools for Software Evolution (SATToSE '14). Springer, 60--78.Google Scholar
- Benwestgarth. Accessed: 2018-04-30. Crossword Sage. https://sourceforge.net/projects/crosswordsage/.Google Scholar
- Buddi. Accessed: 2018-04-30. The Digital Cave. http://buddi.digitalcave.ca.Google Scholar
- Lin Cheng, Jialiang Chang, Zijiang Yang, and Chao Wang. 2016. GUICat: GUI testing as a service. In Proceedings of the International Conference on Automated Software Engineering (ASE '16). ACM, 858--863. Google ScholarDigital Library
- Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet?. In Proceedings of the International Conference on Automated Software Engineering (ASE '16). IEEE Computer Society, 429--440.Google ScholarDigital Library
- Cobertura. Accessed: 2018-04-30. Cobertura. http://cobertura.github.io/cobertura/.Google Scholar
- Svetoslav Ganov, Chip Killmar, Sarfraz Khurshid, and Dewayne E Perry. 2009. Event listener analysis and symbolic execution for testing GUI applications. In Formal Methods and Software Engineering. Springer, 69--87. Google ScholarDigital Library
- Florian Gross, Gordon Fraser, and Andreas Zeller. 2012. Search-based system testing: high coverage, no false alarms. In Proceedings ofthe International Symposium on Software Testing and Analysis (ISSTA '12). ACM, 67--77. Google ScholarDigital Library
- IBM. Accessed: 2018-04-30. Rational Functional Tester. http://www-03.ibm.com/software/products/en/functional.Google Scholar
- Jiri Kovalsky. Accessed: 2018-04-30. Rachota. http://rachota.sourceforge.net/en/index.html.Google Scholar
- Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro. 2012. AutoBlackTest: Automatic Black-Box Testing of Interactive Applications. In Proceedings of the International Conference on Software Testing, Verification and Validation (ICST '12). IEEEComputer Society, 81--90. Google ScholarDigital Library
- Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro. 2014. Automatic testing of GUI-based applications. Software Testing, Verification and Reliability 24, 5 (2014), 341--366. Google ScholarDigital Library
- Leonardo Mariani, Mauro Pezzè, and Daniele Zuddas. 2015. Recent Advances in Automatic Black-Box Testing. In Advances in Computers. Elsevier.Google Scholar
- Leonardo Mariani, Mauro Pezzè, and Daniele Zuddas. 2018. Augusto: Exploiting Popular Functionalities for the Generation of Semantic GUI Tests with Oracles. In Proceedings of the International Conference on Software Engineering (ICSE '18). to appear. Google ScholarDigital Library
- Atif M. Memon, Ishan Banerjee, and Adithya Nagarajan. 2003. GUI Ripping: Reverse Engineering of Graphical User Interfaces for Testing. In Proceedings of The Working Conference on Reverse Engineering (WCRE '03). IEEE Computer Society, 260--269. Google ScholarDigital Library
- Microsoft. Accessed: 2018-04-30. Spec Explorer. https://msdn.microsoft.com/en-us/library/ee620411.aspx.Google Scholar
- Bao N Nguyen, Bryan Robbins, Ishan Banerjee, and Atif Memon. 2014. GUITAR: an innovative tool for automated testing of GUI-driven software. Automated Software Engineering 21, 1 (2014), 65--105. Google ScholarDigital Library
- Cu D. Nguyen, Alessandro Marchetto, and Paolo Tonella. 2013. Automated Oracles: An Empirical Study on Cost and Effectiveness. In Proceedings of the European Software Engineering Conference held jointly with the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE '13). ACM, 136--146. Google ScholarDigital Library
- SeleniumHQ. Accessed: 2018-04-30. Selenium. https://www.seleniumhq.org.Google Scholar
- Adrian Smith. Accessed: 2018-04-30. Universal Password Manager. http://upm.sourceforge.net/index.html.Google Scholar
- Tricentis. Accessed: 2018-04-30. Tosca. https://www.tricentis.com/software-testing-tools/.Google Scholar
- Tanja EJ Vos, Peter M Kruse, Nelly Condori-Fernández, Sebastian Bauersfeld, and Joachim Wegener. 2015. Testar: Tool support for test automation at the user interface level. International Journal of Information System Modeling and Design 6, 3 (2015), 46--83. Google ScholarDigital Library
- Xun Yuan, Myra B Cohen, and Atif M Memon. 2011. GUI Interaction Testing: Incorporating Event Context. IEEE Transactions on Software Engineering 37, 4 (2011), 559--574. Google ScholarDigital Library
Index Terms
- Automatic GUI testing of desktop applications: an empirical assessment of the state of the art
Recommendations
Coverage criteria for GUI testing
A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
Coverage criteria for GUI testing
ESEC/FSE-9: Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineeringA widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
Towards a pattern language for model-based GUI testing
EuroPLoP '14: Proceedings of the 19th European Conference on Pattern Languages of ProgramsGraphical user interfaces (GUIs) have become popular as they appear in everydays' software. GUIs have become an ideal way of interacting with computer programs, making the software friendlier to its users. GUIs have grown, and so has the usage of UI ...
Comments