Atif M. Memon
Mary Lou Soffa
Abstract
Graphical User Interfaces (GUIs) are critical components of today's software. Because GUIs have different characteristics than traditional software, conventional testing techniques do not apply to GUI software. In previous work, we presented an approach to generate GUI test cases, which take the form of sequences of actions. In this paper we develop a test oracle technique to determine if a GUI behaves as expected for a given test case. Our oracle uses a formal model of a GUI, expressed as sets of objects, object properties, and actions. Given the formal model and a test case, our oracle automatically derives the expected state for every action in the test case. We represent the actual state of an executing GUI in terms of objects and their properties derived from the GUI's execution. Using the actual state acquired from an execution monitor, our oracle automatically compares the expected and actual states after each action to verify the correctness of the GUI for the test case. We implemented the oracle as a component in our GUI testing system, called Planning Assisted Tester for grapHical user interface Systems (PATHS), which is based on AI planning. We experimentally evaluated the practicality and effectiveness of our oracle technique and report on the results of experiments to test and verify the behavior of our version of the Microsoft WordPad's GUI.
- 1 L. K. Dillon and Y. S. Ramakrishna. Generating oracles from your favorite temporal logic specifications. In Proceedings of the Fourth ACM SIGSOFT Symposium on the Foundations of Software Engineering, volume 21 of A CM Software Engineering Notes, pages 106-117, New York, Oct.16-18 1996. ACM Press. Google Scholar
- 2 L. K. Dillon and Q. Yu. Oracles for checking temporal properties of concurrent systems. In Proceedings of the ACM SIGSOFT '94Symposium on the Foundations of Software Engineering, pages 140-153, Dec. 1994. Google ScholarDigital Library
- 3 L. du Bousquet, F. Ouabdesselam, J.-L. Richier, and N. Zuanon. Lutess: a specification-driven testing environment for synchronous software. In Proceedings of the 21st International Conference on Software Engineering, pages 267-276. ACM Press, May 1999. Google ScholarDigital Library
- 4 M. Frank, J. J. G. de, D. Gieskeus, and J. D. Foley. Building user interfaces interactively using pre- and postconditions. In Proceedings of CHI '92, 1992. Google ScholarDigital Library
- 5 M. Green. The Design of Graphical User Interfaces. Ph.d. thesis, Department of Computer Science, University of Toronto, 1985. Google ScholarDigital Library
- 6 L. J. Jagadeesan, A. Porter, C. Puchol, J. C. Ramming, and L. G. Votta. Specification-based testing of reactive software: Tools and experiments. In Proceedings of the 19th International Conference on Software Engineering (ICSE '97), pages 525-537, Berlin - Heidelberg - New York, May 1997. Springer. Google ScholarDigital Library
- 7 V. Lifschitz. On the semantics of STRIPS. In M. P. Georgeff and A. L. Lansky, editors, Reasoning about Actions and Plans: Proceedings of the 1986 Workshop, pages 1-9, Timberline, Oregon, June-July 1986. Morgan Kaufmann.Google Scholar
- 8 J. McCarthy. Situations, actions, and causal laws. Memo 2, Stanford University Artificial Intelligence Project, Stanford, California, 1963.Google Scholar
- 9 A. M. Memon, M. E. Pollack, and M. L. Sofia. Using a goal-driven approach to generate test cases for GUIs. In Proceedings of the 21st International Conference on Software Engineering, pages 257-266. ACM Press, May 1999. Google ScholarDigital Library
- 10 B. A. Myers. Why are human-computer interfaces difficult to design and implement? Technical Report CS-93-183, Carnegie Mellon University, School of Computer Science, Google ScholarDigital Library
- 11 B. A. Myers, J. D. Hollan, and I. F. Cruz. Strategic directions in human-computer interaction. A CM Computing Surveys, 28(4):794-809, Dec. 1996. Google ScholarDigital Library
- 12 B. A. Myers, D. R. Olsen, Jr., and J. G. Bonar. User interface tools. In Proceedings of A UM INTERCH{'98 Conference on Human Factors in Computing Systems Adjunct Proceedings, Tutorials, page 239, 1993. Google ScholarDigital Library
- 13 T. Ostrand, A. Anodide, H. Foster, and T. Goradia. A visual test development environment for GUI systems. In Proceedings of the A CM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA-98), volume 23,2 of A UM Software Engineering Notes, pages 82-92, New York, Mar.2-5 1998. ACM Press. Google Scholar
- 14 E. P. D. Pednault. ADL: Exploring the middle ground between STRIPS and the situation calculus. In Proceedings of KR'89, Toronto, Canada, pp 324-331, May 1989. Google ScholarDigital Library
- 15 D. Peters and D. L. Parnas. Generating a test oracle from program documentation. In T. Ostrand, editor, Proceedings of the 1994 International Symposium on Software Testing and Analysis (ISSTA ), pages 58-65, 1994. Google ScholarDigital Library
- 16 D. J. Richardson. TAOS: Testing with analysis and oracle support. In T. Ostrand, editor, Proceedings of the 1994 International Symposium on Software Testing and Analysis (ISSTA): August 17-19, 1994, Seattle, Washington, USA, ACM Sigsoft, pages 138-153, New York, NY 10036, USA, 1994. ACM Press. Google ScholarDigital Library
- 17 D. J. Richardson, S. Leif-Aha, and T. O. OMalley. Specification-based Test Oracles for Reactive Systems. In Proceedings of the 14th International Conference on Software Engineering, pages 105-118, May 1992. Google ScholarDigital Library
- 18 R. K. Shehady and D. P. Siewiorek. A method to automate user interface testing using variable finite state machines. In Proceedings of The Twenty-Seventh Annual International Symposium on Fault-Tolerant Computing (FTCS'97), pages 80-88, Washington - Brussels - Tokyo, June 1997. IEEE Press. Google ScholarDigital Library
- 19 E. Siepman and A. R. Newton. TOBAC: Test Case Browser for Object-Oriented Sofwtare. In Proc. International Symposium on Software Testing and Analysis, pages 154-168, New York, Aug. 1994. ACM Press. Google ScholarDigital Library
- 20 J. Su and P. R. Ritter. Experience in testing the Motif interface. IEEE Software, 8(2):26-33, Mar. 1991. Google ScholarDigital Library
- 21 P. Vogel. An integrated general purpose automated test environment. In T. Ostrand and E. Weyuker, editors, Proceedings of the International Symposium on Software Testing and Analysis, pages 61-69, New York, NY, USA, June 1993. ACM Press. Google ScholarDigital Library
- 22 D. S. Weld. An introduction to least commitment plann:ng. AI Magazine, 15(4):27-61, 1994.Google Scholar
- 23 D. S. Weld. Recent advances in AI planning. AI Magazine, 20(1):55-64, Spring 1999.Google Scholar
- 24 W. I. Wittel, Jr. and T. G. Lewis. Integrating the MVC paradigm into an object-oriented framework to accelerate GUI application development. Technical Report 91-60-06, Department of Computer Science, Oregon State University, 1991.July 1993. Google ScholarDigital Library
Index Terms
- Automated test oracles for GUIs
Recommendations
Designing and comparing automated test oracles for GUI-based software applications
Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This article shows that the test oracle, a mechanism that determines whether a ...
Automated test oracles for GUIs
SIGSOFT '00/FSE-8: Proceedings of the 8th ACM SIGSOFT international symposium on Foundations of software engineering: twenty-first century applicationsGraphical User Interfaces (GUIs) are critical components of today's software. Because GUIs have different characteristics than traditional software, conventional testing techniques do not apply to GUI software. In previous work, we presented an approach ...
Regression testing of GUIs
Although graphical user interfaces (GUIs) constitute a large part of the software being developed today and are typically created using rapid prototyping, there are no effective regression testing techniques for GUIs. The needs of GUI regression testing ...
Comments