Qing Xie
Skip Abstract Section
Abstract
Graphical user interfaces (GUIs) are one of the most commonly used parts of today's software. Despite their ubiquity, testing GUIs for functional correctness remains an understudied area. A typical GUI gives many degrees of freedom to an end-user, leading to an enormous input event interaction space that needs to be tested. GUI test designers generate and execute test cases (modeled as sequences of user events) to traverse its parts; targeting a subspace in order to maximize fault detection is a nontrivial task. In this vein, in previous work, we used informal GUI code examination and personal intuition to develop an event-interaction graph (EIG). In this article we empirically derive the EIG model via a pilot study, and the resulting EIG validates our intuition used in previous work; the empirical derivation process also allows for model evolution as our understanding of GUI faults improves. Results of the pilot study show that events interact in complex ways; a GUI's response to an event may vary depending on the context established by preceding events and their execution order. The EIG model helps testers to understand the nature of interactions between GUI events when executed in test cases and why certain events detect faults, so that they can better traverse the event space. New test adequacy criteria are defined for the EIG; new algorithms use these criteria and EIG to systematically generate test cases that are shown to be effective on four fielded open-source applications.
- Andrews, J. H., Briand, L. C., and Labiche, Y. 2005. Is mutation an appropriate tool for testing experiments? In ICSE '05: Proceedings of the 27th International Conference on Software Engineering. ACM Press, New York, NY, 402--411. Google Scholar
Digital Library
- Auguston, M., Michael, J. B., and Shing, M.-T. 2005. Environment behavior models for scenario generation and testing automation. In A-MOST '05: Proceedings of the 1st International Workshop on Advances in Model-Based Testing. ACM Press, New York, NY, 1--6. Google ScholarDigital Library
- Bernhard, P. J. 1994. A reduced test suite for protocol conformance testing. ACM Trans. Softw. Eng. Methodol. 3, 3 (July), 201--220. Google ScholarDigital Library
- Chow, T. S. 1978. Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. SE-4, 3, 178--187. Google ScholarDigital Library
- Clarke, J. M. 1998. Automated test generation from a behavioral model. In Proceedings of the Pacific Northwest Software Quality Conference (Pnsqc/Pacific Agenda, Portland, OR).Google Scholar
- Esmelioglu, S. and Apfelbaum, L. 1997. Automated test generation, execution, and reporting. In Proceedings of the Pacific Northwest Software Quality Conference (Pnsqc/Pacific Agenda, Portland, OR). 127--142.Google Scholar
- Harrold, M. J., Offut, A. J., and Tewary, K. 1997. An approach to fault modelling and fault seeding using the program dependence graph. J. Syst. Softw. 36, 3 (Mar.), 273--296. Google ScholarDigital Library
- Hicinbothom, J. H. and Zachary, W. W. 1993. A tool for automatically generating transcripts of human-computer interaction. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting. Special Sessions: Demonstrations, vol. 2. Human Factors and Ergonomics Society, Santa Monica, CA, 1042.Google Scholar
- Howe, A., von Mayrhauser, A., and Mraz, R. T. 1997. Test case generation as an AI planning problem. Automat. Softw. Eng. 4, 77--106. Google ScholarDigital Library
- Imanian, J. A. 2005. Automatic test case generation for reactive software systems based on environment models. Ph.D. dissertation. Naval Postgraduate School, Monterey, CA.Google Scholar
- Jorgensen, P. C. and Erickson, C. 1994. Object-oriented integration testing. Commun. ACM 37, 9, 30--38. Google ScholarDigital Library
- Kasik, D. J. and George, H. G. 1996. Toward automatic generation of novice user test scripts. In Proceedings of the Conference on Human Factors in Computing Systems: Common Ground. ACM Press, New York, NY, 244--251. Google ScholarDigital Library
- Leow, W. K., Khoo, S. C., and Sun, Y. 2004. Automated generation of test programs from closed specifications of classes and test cases. In ICSE '04: Proceedings of the 26th International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos, CA, 96--105. Google ScholarDigital Library
- Maurer, P. M. 1990. Generating test data with enhanced context-free grammars. IEEE Softw. 7, 4 (July), 50--55. Google ScholarDigital Library
- Memon, A., Banerjee, I., and Nagarajan, A. 2003. GUI Ripping: Reverse engineering of graphical user interfaces for testing. In WCRE '03: Proceedings of the 10th Working Conference on Reverse Engineering. IEEE Computer Society Press, Los Alamitos, CA, 260--269. Google ScholarDigital Library
- Memon, A. M. 2001. A comprehensive framework for testing graphical user interfaces. Ph.D. dissertation. Department of Computer Science, University of Pittsburgh, Pittsburgh, PA. Google ScholarDigital Library
- Memon, A. M. 2002. GUI testing: Pitfalls and process. IEEE Comput. 35, 8 (Aug.), 90--91. Google ScholarDigital Library
- Memon, A. M. 2003. Advances in GUI testing. In Advances in Computers, 58, M. V. Zelkowitz, Ed. Elsevier, Amsterdam, The Netherlands, 150--203.Google Scholar
- Memon, A. M., Nagarajan, A., and Xie, Q. 2005. Automating regression testing for evolving GUI software. J. Softw. Maint. 17, 1, 27--64. Google ScholarDigital Library
- Memon, A. M., Pollack, M. E., and Soffa, M. L. 1999. Using a goal-driven approach to generate test cases for GUIs. In ICSE '99: Proceedings of the 21st International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos, CA, 257--266. Google ScholarDigital Library
- Memon, A. M., Pollack, M. E., and Soffa, M. L. 2000. Automated test oracles for GUIs. In SIGSOFT '00/FSE-8: Proceedings of the 8th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York, NY, 30--39. Google ScholarDigital Library
- Memon, A. M., Pollack, M. E., and Soffa, M. L. 2001a. Hierarchical GUI test case generation using automated planning. IEEE Trans. Softw. Eng. 27, 2 (Feb.), 144--155. Google ScholarDigital Library
- Memon, A. M. and Soffa, M. L. 2003. Regression testing of GUIs. In ESEC/FSE-11: Proceedings of the 9th European Software Engineering Conference Held Jointly with the 11th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York, NY, 118--127. Google ScholarDigital Library
- Memon, A. M., Soffa, M. L., and Pollack, M. E. 2001b. Coverage criteria for GUI testing. In ESEC/FSE-9: Proceedings of the 8th European Software Engineering Conference Held Jointly with the 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York, NY, 256--267. Google ScholarDigital Library
- Memon, A. M. and Xie, Q. 2005. Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software. IEEE Trans. Softw. Eng. 31, 10 (Oct.), 884--896. Google ScholarDigital Library
- Offutt, A. J. and Hayes, J. H. 1996. A semantic model of program faults. In ISSTA '96: Proceedings of the 1996 ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM Press, New York, NY, 195--200. Google ScholarDigital Library
- Rapps, S. and Weyuker, E. J. 1982. Data flow analysis techniques for test data selection. In ICSE '82: Proceedings of the 6th International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos, CA, 272--278. Google ScholarDigital Library
- Scheetz, M., von Mayrhauser, A., France, R., Dahlman, E., and Howe, A. E. 1999. Generating test cases from an OO model with an AI planning system. In Proceedings of the 10th International Symposium on Software Reliability Engineering. IEEE Computer Society Press, Los Alamitos, CA, 250--259. Google ScholarDigital Library
- Shehady, R. K. and Siewiorek, D. P. 1997. A method to automate user interface testing using variable finite state machines. In Proceedings of the 27th Annual International Symposium on Fault-Tolerant Computing (FTCS'97). IEEE Computer Society Press, Los Alamitos, CA, 80--88. Google ScholarDigital Library
- Spacco, J., Hovemeyer, D., and Pugh, W. 2006. Tracking defect warnings across versions. In MSR '06: Proceedings of the 2006 International Workshop on Mining Software Repositories. ACM Press, New York, NY, 133--136. Google ScholarDigital Library
- von Mayrhauser, A. and Crawford-Hines, S. 1993. Automated testing support for a robot tape library. In Proceedings of the 4th International Software Reliability Engineering Conference. IEEE Computer Society Press, Los Alamitos, CA, 6--14.Google Scholar
- von Mayrhauser, A., Mraz, R. T., and Walls, J. 1994. Domain based regression testing. In Proceedings of the International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 26--35. Google ScholarDigital Library
- White, L. and Almezen, H. 2000. Generating test cases for GUI responsibilities using complete interaction sequences. In ISSRE '00: Proceedings of the 11th International Symposium on Software Reliability Engineering. IEEE Computer Society Press, Los Alamitos, CA, 110. Google ScholarDigital Library
- Whittaker, J. A. 1992. Markov chain techniques for software testing and reliability analysis. Ph.D. dissertation. University of Tennessee, Knoxville, TN. Google ScholarDigital Library
- Whittaker, J. A. and Thomason, M. G. 1994. A Markov chain model for statistical software testing. IEEE Trans. Softw. Eng. 20, 10, 812--824. Google ScholarDigital Library
- Woit, D. 1998. Conditional-event usage testing. In CASCON '98: Proceedings of the 1998 Conference of the Centre for Advanced Studies on Collaborative Research. IBM Press, Indianapolis, IN, 23. Google ScholarDigital Library
- Woit, D. M. 1993. Specifying operational profiles for modules. In ISSTA '93: Proceedings of the 1993 ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM Press, New York, NY, 2--10. Google ScholarDigital Library
- Xie, Q. and Memon, A. M. 2005. Rapid “crash testing” for continuously evolving GUI-based software applications. In ICSM '05: Proceedings of the 21st IEEE International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 473--482. Google ScholarDigital Library
- Xie, Q. and Memon, A. M. 2006. Automated model-based testing of community-driven open source GUI applications. In ICSM '06: Proceedings of the 22nd IEEE International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 145--154. Google ScholarDigital Library
- Xie, T., Marinov, D., Schulte, W., and Notkin, D. 2005. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 05). Spinger, Berlin, Germany, 365--381. Google ScholarDigital Library
- Zeller, A. 2002. Isolating cause-effect chains from computer programs. ACM SIGSOFT Softw. Eng. Notes 27, 6, 1--10. Google ScholarDigital Library
Index Terms
- Using a pilot study to derive a GUI model for automated testing
Recommendations
Designing and comparing automated test oracles for GUI-based software applications
Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This article shows that the test oracle, a mechanism that determines whether a ...
An Extensible Heuristic-Based Framework for GUI Test Case Maintenance
ICSTW '09: Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation WorkshopsGraphical user interfaces (GUIs) make up a large portion of the code comprising many modern software applications. However, GUI testing differs significantly from testing of traditional software. One respect in which this is true is test case ...
GUI Interaction Testing: Incorporating Event Context
Graphical user interfaces (GUIs), due to their event-driven nature, present an enormous and potentially unbounded way for users to interact with software. During testing, it is important to “adequately cover” this interaction space. In this paper, we ...
Comments