ABSTRACT
Existing web test generators derive test paths from a navigational model of the web application, completed with either manually or randomly generated input values. However, manual test data selection is costly, while random generation often results in infeasible input sequences, which are rejected by the application under test. Random and search-based generation can achieve the desired level of model coverage only after a large number of test execution at- tempts, each slowed down by the need to interact with the browser during test execution. In this work, we present a novel web test generation algorithm that pre-selects the most promising candidate test cases based on their diversity from previously generated tests. As such, only the test cases that explore diverse behaviours of the application are considered for in-browser execution. We have implemented our approach in a tool called DIG. Our empirical evaluation on six real-world web applications shows that DIG achieves higher coverage and fault detection rates significantly earlier than crawling-based and search-based web test generators.
- Nadia Alshahwan and Mark Harman. 2012. Augmenting test suites effectiveness by increasing output diversity. In 34th International Conference on Software Engineering, ICSE. 1345–1348. Google ScholarDigital Library
- Andrea Arcuri. 2013. It really does matter how you normalize the branch distance in search-based software testing. Software Testing, Verification and Reliability 23, 2 (2013), 119–147.Google ScholarCross Ref
- Shay Artzi, Julian Dolby, Simon Holm Jensen, Anders Møller, and Frank Tip. 2011. A Framework for Automated Testing of Javascript Web Applications. In Proceedings of the 33rd International Conference on Software Engineering (ICSE ’11). ACM, New York, NY, USA, 571–580. Google ScholarDigital Library
- Kartik Bajaj, Karthik Pattabiraman, and Ali Mesbah. 2015. Synthesizing Web Element Locators. In Proceedings of 30th IEEE/ACM International Conference on Automated Software Engineering (ASE ’15). IEEE Computer Society, 331–341.Google ScholarDigital Library
- Matteo Biagiola, Filippo Ricca, and Paolo Tonella. 2017. Search Based Path and Input Data Generation for Web Application Testing. In International Symposium on Search Based Software Engineering. Springer, 18–32.Google Scholar
- Matteo Biagiola, Andrea Stocco, Ali Mesbah, Filippo Ricca, and Paolo Tonella. 2019. Web Test Dependency Detection. In Proceedings of 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019). ACM, 12 pages. Google ScholarDigital Library
- Robert V. Binder. 1996. Testing object-oriented software: a survey. Software Testing, Verification and Reliability 6, 3-4 (1996), 125–252.Google ScholarCross Ref
- Tsong Yueh Chen, Fei-Ching Kuo, Robert G Merkel, and TH Tse. 2010. Adaptive random testing: The art of test case diversity. Journal of Systems and Software 83, 1 (2010), 60–66. Google ScholarDigital Library
- Tsong Yueh Chen, Hing Leung, and IK Mak. 2004. Adaptive random testing. In Annual Asian Computing Science Conference. Springer, 320–329.Google ScholarDigital Library
- Ilinca Ciupa, Andreas Leitner, Manuel Oriol, and Bertrand Meyer. 2008. ARTOO: adaptive random testing for object-oriented software. In Proceedings of the 30th international conference on Software engineering. ACM, 71–80. Google ScholarDigital Library
- Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2001. Introduction to Algorithms (2nd ed.). MIT Press and McGraw-Hill. Google ScholarDigital Library
- dimeshift 2018. DimeShift: easiest way to track your expenses. https://github. com/jeka-kiselyov/dimeshift. (2018).Google Scholar
- Robert Feldt, Simon M. Poulding, David Clark, and Shin Yoo. 2016. Test Set Diameter: Quantifying the Diversity of Sets of Test Cases. In 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, Chicago, IL, USA, April 11-15, 2016. 223–233.Google Scholar
- Mark Fewster and Dorothy Graham. 1999. Software Test Automation: Effective Use of Test Execution Tools. Addison-Wesley Longman Publishing Co., Inc. Google ScholarDigital Library
- Martin Fowler. 2013. PageObject. http://martinfowler.com/bliki/PageObject.html. (2013). Accessed: 2018-08-01.Google Scholar
- Gordon Fraser and Andrea Arcuri. 2011. Evolutionary Generation of Whole Test Suites. In 11 t h International Conference on Quality Software (QSIC), Manuel Núñez, Robert M. Hierons, and Mercedes G. Merayo (Eds.). IEEE Computer Society, Madrid, Spain, 31–40. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation. IEEE Transactions on Software Engineering 39, 2 (feb. 2013), 276 –291. Google ScholarDigital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed automated random testing. In Programming Language Design and Implementation (PLDI 2005), Vivek Sarkar and Mary W. Hall (Eds.). ACM, 213–223. Google ScholarDigital Library
- Mouna Hammoudi, Gregg Rothermel, and Andrea Stocco. 2016. WATERFALL: An Incremental Approach for Repairing Record-replay Tests of Web Applications. In Proceedings of 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE ’16). ACM, 751–762. Google ScholarDigital Library
- Istanbul 2018. Istanbul: JavaScript test coverage made simple. https://istanbul.js. org. (2018). Accessed: 2018-08-01.Google Scholar
- JS-frameworks 2018. Front-end JavaScript frameworks. https://github.com/ collections/front-end-javascript-frameworks. (2018).Google Scholar
- O. Koresteleva. 2004. Nonparametric Methods in Statistics with SAS Applications. CRC Press, Boca Raton, FL.Google Scholar
- Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2016. Approaches and Tools for Automated End-to-End Web Testing. Advances in Computers 101 (2016), 193–237.Google ScholarCross Ref
- Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2015. Using Multi-Locators to Increase the Robustness of Web Test Cases. In Proceedings of 8th IEEE International Conference on Software Testing, Verification and Validation (ICST ’15). IEEE, 1–10.Google ScholarCross Ref
- Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2016. ROBULA+: An Algorithm for Generating Robust XPath Locators for Web Testing. Journal of Software: Evolution and Process (2016), 28:177–204. Google ScholarDigital Library
- Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2018. PESTO: Automated migration of DOM-based Web tests towards the visual approach. Software Testing, Verification And Reliability 28, 4 (2018).Google Scholar
- Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, Vol. 10. 707–710.Google Scholar
- Yu Lin, Xucheng Tang, Yuting Chen, and Jianjun Zhao. 2009. A divergenceoriented approach to adaptive random testing of Java programs. In Automated Software Engineering, 2009. ASE’09. 24th IEEE/ACM International Conference on. IEEE, 221–232. Google ScholarDigital Library
- Sean Luke. 2013. Essentials of Metaheuristics (second ed.). Lulu. Available for free at http://cs.gmu.edu/∼sean/book/metaheuristics/.Google Scholar
- Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. EvoDroid: Segmented Evolutionary Testing of Android Apps. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 599–609. Google ScholarDigital Library
- Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective Automated Testing for Android Applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016). ACM, New York, NY, USA, 94–105. Google ScholarDigital Library
- Alessandro Marchetto, Paolo Tonella, and Filippo Ricca. 2008. State-Based Testing of Ajax Web Applications. In Proceedings of the First International Conference on Software Testing, Verification, and Validation (ICST). 121–130. Google ScholarDigital Library
- Alessandro Marchetto, Paolo Tonella, and Filippo Ricca. 2008. State-Based Testing of Ajax Web Applications. In Proceedings of the 2008 International Conference on Software Testing, Verification, and Validation (ICST ’08). IEEE Computer Society, Washington, DC, USA, 121–130. Google ScholarDigital Library
- Phil McMinn. 2004. Search-based software test data generation: a survey. Softw. Test., Verif. Reliab. 14, 2 (2004), 105–156. Google ScholarDigital Library
- Ali Mesbah and Arie van Deursen. 2009. Invariant-based Automatic Testing of AJAX User Interfaces. In Proceedings of the 31st International Conference on Software Engineering (ICSE ’09). IEEE Computer Society, Washington, DC, USA, 210–220. Google ScholarDigital Library
- Ali Mesbah, Arie van Deursen, and Stefan Lenselink. 2012. Crawling Ajax-Based Web Applications through Dynamic Analysis of User Interface State Changes. ACM Transactions on the Web (TWEB) 6, 1 (2012), 3:1–3:30. Google ScholarDigital Library
- Ali Mesbah, Arie van Deursen, and Danny Roest. 2012. Invariant-Based Automatic Testing of Modern Web Applications. IEEE Trans. Software Eng. 38, 1 (2012), 35– 53. Google ScholarDigital Library
- Amin Milani Fard and Ali Mesbah. 2013. Feedback-directed Exploration of Web Applications to Derive Test Models. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE). IEEE Computer Society, 278–287. http://www.ece.ubc.ca/~amesbah/docs/issre13.pdfGoogle ScholarCross Ref
- Frolin S Ocariza Jr, Karthik Pattabiraman, and Ali Mesbah. 2017. Detecting unknown inconsistencies in web applications. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 566–577. Google ScholarDigital Library
- Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In Proceedings of the 29th International Conference on Software Engineering (ICSE ’07). IEEE Computer Society, Washington, DC, USA, 75–84. Google ScholarDigital Library
- pagekit 2018. Pagekit: modular and lightweight CMS. https://github.com/pagekit/ pagekit. (2018).Google Scholar
- Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2018. Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets. IEEE Trans. Software Eng. 44, 2 (2018), 122–158.Google ScholarCross Ref
- PetClinic 2018. Angular version of the Spring PetClinic web application. https: //github.com/spring-petclinic/spring-petclinic-angular. (2018).Google Scholar
- phoenix 2018. Phoenix: Trello tribute done in Elixir, Phoenix Framework, React and Redux. https://github.com/bigardone/phoenix-trello. (2018).Google Scholar
- retroboard 2018. Retrospective Board. https://github.com/antoinejaussoin/retroboard. (2018).Google Scholar
- Selenium 2018. SeleniumHQ Web Browser Automation. http://www.seleniumhq. org. (2018). Accessed: 2018-08-01.Google Scholar
- Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: a concolic unit testing engine for C. In 10 t h European Software Engineering Conference and 13th ACM International Symposium on Foundations of Software Engineering (ESEC/FSE ’05), Michel Wermelinger and Harald Gall (Eds.). ACM, 263–272. Google ScholarDigital Library
- splittypie 2018. Splittypie: easy expense splitting. https://github.com/cowbell/ splittypie. (2018).Google Scholar
- Andrea Stocco, Maurizio Leotta, Filippo Ricca, and Paolo Tonella. 2016. Clustering-Aided Page Object Generation for Web Testing. In Proceedings of 16th International Conference on Web Engineering (ICWE ’16). Springer, 132–151.Google ScholarCross Ref
- Andrea Stocco, Maurizio Leotta, Filippo Ricca, and Paolo Tonella. 2017. APOGEN: Automatic Page Object Generator for Web Testing. Software Quality Journal 25, 3 (Sept. 2017), 1007–1039. Google ScholarDigital Library
- Andrea Stocco, Rahulkrishna Yandrapally, and Ali Mesbah. 2018. Visual Web Test Repair. In Proceedings of the 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’18). ACM. Google ScholarDigital Library
- tool 2019. DIG: Diversity-based E2E web test generator. https://github.com/ matteobiagiola/FSE19-submission-material-DIG. (2019).Google Scholar
- Arie van Deursen. 2015. Beyond Page Objects: Testing Web Applications with State Objects. ACM Queue 13, 6 (2015), 20. ESEC/FSE ’19, August 26–30, 2019, Tallinn, Estonia Matteo Biagiola, Andrea Stocco, Filippo Ricca, Paolo Tonella Google ScholarDigital Library
- Arie van Deursen. 2015. Testing Web Applications with State Objects. Commun. ACM 58, 8 (July 2015), 36–43. Google ScholarDigital Library
- Rahulkrishna Yandrapally, Suresh Thummalapenta, Saurabh Sinha, and Satish Chandra. 2014. Robust Test Automation Using Contextual Clues. In Proceedings of 2014 International Symposium on Software Testing and Analysis (ISSTA ’14). ACM, 304–314. Google ScholarDigital Library
- Bing Yu, Lei Ma, and Cheng Zhang. 2015. Incremental Web Application Testing Using Page Object. In Proceedings of the 2015 Third IEEE Workshop on Hot Topics in Web Systems and Technologies (HotWeb) (HOTWEB ’15). IEEE Computer Society, Washington, DC, USA, 1–6. Google ScholarDigital Library
- Xun Yuan and Atif M. Memon. 2007. Using GUI Run-Time State as Feedback to Generate Test Cases. In ICSE ’07: Proceedings of the 29th International Conference on Software Engineering. IEEE Computer Society, Washington, DC, USA, 396–405. Google ScholarDigital Library
Index Terms
- Diversity-based web test generation
Recommendations
Web test dependency detection
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringE2E web test suites are prone to test dependencies due to the heterogeneous multi-tiered nature of modern web apps, which makes it difficult for developers to create isolated program states for each test case. In this paper, we present the first ...
Test case generation based on mutations over user execution traces
AbstractAutomatic test case generation is usually based on models of the software under test. However, those models may not exist or may be outdated and so, the test case generation must resort to other artifacts. In a software maintenance context, test ...
Automated generation of visual web tests from DOM-based web tests
SAC '15: Proceedings of the 30th Annual ACM Symposium on Applied ComputingFunctional test automation is increasingly adopted by web applications developers. In particular, 2nd generation tools overcome the limitations of 1st generation tools, based on screen coordinates, by providing APIs for easy selection and interaction ...
Comments