Yogesh Singh
Skip Abstract Section
Abstract
What is the importance of automated test data generation? This question is of concern for the entire software community. We want to know the meaning and usefulness of automated test data generation. Is it related to design of test cases or execution of already designed test cases or both? How can we generate expected output using automated test data generation technique? This is a weak area and limits the applicability of automated testing. The test case creation, execution and evaluation are three distinct activities. The execution and evaluation of test cases can be easily automated, but creation needs expected output that is difficult to automate due to its dependency on functionality of the program. This paper addresses the significance of techniques and tools available for automated test data generation. In this work, we analyze and assess the existing techniques for automated expected output generation and discuss the practical significance of these techniques.
- Y. Singh. 2012. Software Testing. Cambridge University Press, USA.Google Scholar
- K. Lakhotia, P. McMinn, and M. Harman. 2009. Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet?. In Proceedings of the 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques (TAIC-PART '09). IEEE Computer Society, Washington, DC, USA, 95--104. DOI=10.1109/TAICPART.2009.15 http://dx.doi.org/10.1109/TAICPART.2009.15. Google ScholarDigital Library
- C. V. Ramamoorthy, S. -B. F. Ho, and W. T. Chen. 1976. On the Automated Generation of Program Test Data. IEEE Trans. Softw. Eng. 2, 4 (July 1976), 293--300. DOI=10.1109/TSE.1976.233835 http://dx.doi.org/10.1109/TSE.1976.233835. Google ScholarDigital Library
- John B. Goodenough. 1975. Toward a theory of test data selection. IEEE Trans. Softw. Eng. 1, 1 (March 1975), 156--173. DOI=10.1109/TSE.1975.6312836 http://dx.doi.org/10.1109/TSE.1975.6312836. Google ScholarDigital Library
- S. Anand, Edmund K. Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, and Phil Mcminn. 2013. An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86, 8 (August 2013), 1978--2001. DOI=10.1016/j.jss.2013.02.061 http://dx.doi.org/10.1016/j.jss.2013.02.061. Google ScholarDigital Library
- B. Korel. 1996. Automated test data generation for programs with procedures. In Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis (ISSTA '96), Steve J. Zeil and Will Tracz (Eds.). ACM, New York, NY, USA, 209--215. DOI=10.1145/229000.226319 http://doi.acm.org/10.1145/229000.226319. Google ScholarDigital Library
- A.V.K. Shanthi and G. Mohan Kumar. 2011. Automated test cases generation for object oriented software. Indian Journal of Computer Science & Engineering 2, 4, Aug-Sept, 543--546.Google Scholar
- A. Baresel, H. Sthamer, and M. Schmidt. 2002. Fitness function design to improve evolutionary structural testing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2002), New York, USA, Morgan Kaufmann, 1329--1336. Google ScholarDigital Library
- B. Jones, H. Sthamer, X. Yang, and D. Eyres. 1995. The automatic generation of software test data sets using adaptive search techniques. In Proceedings of the 3rd International Conference on Software Quality Management, Seville, Spain, 435--444.Google Scholar
- Joachim Wegener, Kerstin Buhr, and Hartmut Pohlheim. 2002. Automatic Test Data Generation For Structural Testing Of Embedded Software Systems By Evolutionary Testing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '02), William B. Langdon, Erick Cantí-Paz, Keith E. Mathias, Rajkumar Roy, David Davis, Riccardo Poli, Karthik Balakrishnan, Vasant G. Honavar, Günter Rudolph, Joachim Wegener, Larry Bull, Mitchell A. Potter, Alan C. Schultz, J. F. Miller, E. Burke, and Natasa Jonoska (Eds.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1233--1240 Google ScholarDigital Library
- V. Santiago, A.S. Martins do Amaral, N. L. Vijaykumar, Maria de Fatima Mattiello-Francisco, E. Martins, and Odnei Cuesta Lopes. 2006. A Practical Approach for Automated Test Case Generation using Statecharts. In Proceedings of the 30th Annual International Computer Software and Applications Conference - Volume 02 (COMPSAC '06), Vol. 2. IEEE Computer Society, Washington, DC, USA, 183--188. DOI=10.1109/COMPSAC.2006.100 http://dx.doi.org/10.1109/COMPSAC.2006.100. Google ScholarDigital Library
- R.A. DeMillo and A. J. Offutt. 1991. Constraint-Based Automatic Test Data Generation. IEEE Trans. Softw. Eng. 17, 9 (September 1991), 900--910. DOI=10.1109/32.92910 http://dx.doi.org/10.1109/32.92910. Google ScholarDigital Library
- M. Balcer, W. Hasling, and T. Ostrand. 1989. Automatic generation of test scripts from formal test specifications. SIGSOFT Softw. Eng. Notes 14, 8 (November 1989), 210--218. DOI=10.1145/75309.75332 http://doi.acm.org/10.1145/75309.75332. Google ScholarDigital Library
- R. Malhotra, A. Kaur and Y. Singh. 2010. A Regression Test Selection and Prioritization Technique. Journal of Information Processing Systems 6, 2 (June 2010), 235--252.Google ScholarCross Ref
- Robert M. Poston. 1997. Automating Specification-Based Software Testing (1st ed.). IEEE Computer Society Press, Los Alamitos, CA, USA. Google ScholarDigital Library
- Robert M. Poston. 1994. Automated testing from object models. Commun. ACM 37, 9 (September 1994), 48--58. DOI=10.1145/182987.184074 http://doi.acm.org/10.1145/182987.184074. Google ScholarDigital Library
- Q. Xie and A.M. Memon. 2007. Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. Softw. Eng. Methodol. 16, 1, Article 4 (February 2007). DOI=10.1145/1189748.1189752 http://doi.acm.org/10.1145/1189748.1189752. Google ScholarDigital Library
- Dennis Peters and David L. Parnas. 1994. Generating a test oracle from program documentation: work in progress. In Proceedings of the 1994 ACM SIGSOFT international symposium on Software testing and analysis (ISSTA '94), Thomas Ostrand (Ed.). ACM, New York, NY, USA, 58--65. DOI=10.1145/186258.186508 http://doi.acm.org/10.1145/186258.186508. Google ScholarDigital Library
- Debra J. Richardson. 1994. TAOS: Testing with Analysis and Oracle Support. In Proceedings of the 1994 ACM SIGSOFT international symposium on Software testing and analysis (ISSTA '94), Thomas Ostrand (Ed.). ACM, New York, NY, USA, 138--153. DOI=10.1145/186258.187158 http://doi.acm.org/10.1145/186258.187158. Google ScholarDigital Library
- L. K. Dillon and Y. S. Ramakrishna. 1996. Generating oracles from your favorite temporal logic specifications. SIGSOFT Softw. Eng. Notes 21, 6 (October 1996), 106--117. DOI=10.1145/250707.239116 http://doi.acm.org/10.1145/250707.239116. Google ScholarDigital Library
- P. A. V. Hall. 1998. Towards testing with respect to formal specification. In Second IEE/BCS Conference on Software Engineering IEEE, 159--163.Google Scholar
- P. A. V. Hall. 1991. Relationship between specifications and testing. Inf. Softw. Technol. 33, 1 (February 1991), 47-52. DOI=10.1016/0950-5849(91)90023-5 http://dx.doi.org/10.1016/0950-5849(91)90023-5. Google ScholarDigital Library
- Debra J. Richardson, Stephanie Leif Aha, and T. Owen O'Malley. 1992. Specification-based test oracles for reactive systems. In Proceedings of the 14th international conference on Software engineering (ICSE '92). ACM, New York, NY, USA, 105--118. DOI=10.1145/143062.143100 http://doi.acm.org/10.1145/143062.143100. Google ScholarDigital Library
- L. I. Manolache and D. G. Kourie. 2001. Software testing using model programs. Softw. Pract. Exper. 31, 13 (October 2001), 1211--1236. DOI=10.1002/spe.409 http://dx.doi.org/10.1002/spe.409. Google ScholarDigital Library
- Patrick J. Schroeder and Bogdan Korel. 2000. Black-box test reduction using input-output analysis.SIGSOFT Softw. Eng. Notes 25, 5 (August 2000), 173--177. DOI=10.1145/347636.349042 http://doi.acm.org/10.1145/347636.349042. Google ScholarDigital Library
- G. Di Lucca, A. Fasolino, and F. Faralli. 2002. Testing Web Applications. In Proceedings of the International Conference on Software Maintenance (ICSM'02) (ICSM '02). IEEE Computer Society, Washington, DC, USA, 310--319. Google ScholarDigital Library
- K. K. Aggarwal, Yogesh Singh, A. Kaur, and O. P. Sangwan. 2004. A neural net based approach to Test Oracle. SIGSOFT Softw. Eng. Notes 29, 3 (May 2004), 1--6. DOI=10.1145/986710.986725 http://doi.acm.org/10.1145/986710.986725. Google ScholarDigital Library
- Ye Mao, Feng Boqin, Zhu Li, and Lin Yao. 2006. Neural networks based automated test oracle for software testing. In Proceedings of the 13th international conference on Neural information processing - Volume Part III (ICONIP'06), Irwin King, Laiwan Chan, Jun Wang, and DeLiang Wang (Eds.), Vol. Part III. Springer-Verlag, Berlin, Heidelberg, 498--507. Google ScholarDigital Library
- M. Vanmali, M. Last, and A. Kandel. 2002. Using a neural network in the software testing process. International Journal of Intelligent Systems 17, 1, 45--62.Google ScholarCross Ref
- Atif M. Memon, Martha E. Pollack, and Mary Lou Soffa. 2000. Automated test oracles for GUIs. SIGSOFT Softw. Eng. Notes 25, 6 (November 2000), 30--39. DOI=10.1145/357474.355050 http://doi.acm.org/10.1145/357474.355050. Google ScholarDigital Library
- S.R. Shahamiri, W.M.N.W. Kadir, S.Z. Mohd-Hashim, 2009. A comparative study on automated software test oracle methods, in Proceedings of the 2009 Fourth International Conference on Software Engineering Advances, IEEE Computer Society, Porto, Portugal, 140--145. Google ScholarDigital Library
- E.T. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5, 507--525.Google ScholarDigital Library
- J. Su and P. R. Ritter. 1991. Experience in testing the Motif interface. IEEE Software 8, 2 (Mar. 1991), 6--33. Google ScholarDigital Library
- P. Vogel. 1993. An integrated general purpose automated test environment. In T. Ostrand and E. Weyuker, editors, Proceedings of the International Symposium on Software Testing and Analysis, New York, NY, USA, ACM Press, 61--69. Google ScholarDigital Library
Recommendations
Symbolic execution and program testing
This paper describes the symbolic execution of programs. Instead of supplying the normal inputs to a program (e.g. numbers) one supplies symbols representing arbitrary values. The execution proceeds as in a normal execution except that values may be ...
Symbolic execution for software testing: three decades later
The challenges---and great promise---of modern symbolic execution techniques, and the tools to help implement them.
Comments