Antonia Bertolino
Pietro Braione
Guglielmo De Angelis
Leonardo Mariani
Mauro Pezzè
Stefano Russo
Paolo Tonella
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this article, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements, and how field tests are managed, and identify many challenging research directions.
Supplemental Material
Available for Download
zip
bertolino.zip (167.2 KB)
Supplemental movie, appendix, image and software files for, A Survey of Field-based Testing Techniques
- Mohammed Akour, Akanksha Jaidev, and Tariq M. King. 2011. Towards change propagating test models in autonomic and adaptive systems. In Proceedings of the 18th International Conference on the Engineering of Computer-Based Systems (ECBS’11). IEEE, 89--96. DOI:https://doi.org/10.1109/ECBS.2011.23Google Scholar
- Midhat Ali, Francesco De Angelis, Daniele Fanì, Antonia Bertolino, Guglielmo De Angelis, and Andrea Polini. 2014. An extensible framework for online testing of choreographed services. Computer 47, 2 (Feb. 2014), 23--29. DOI:https://doi.org/10.1109/MC.2013.407Google ScholarDigital Library
- Khaled Alnawasreh, Patrizio Pelliccione, Zhenxiao Hao, Mårten Rånge, and Antonia Bertolino. 2017. Online robustness testing of distributed embedded systems: An industrial approach. In Proceedings of the IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP’17). IEEE, 133--142. DOI:https://doi.org/10.1109/ICSE-SEIP.2017.17Google ScholarDigital Library
- Sultan Alyahya. 2020. Crowdsourced software testing: A systematic literature review. Inf. Softw. Technol. 127, Article 106363 (Nov. 2020). DOI:https://doi.org/10.1016/j.infsof.2020.106363Google Scholar
- Xiaoying Bai, Shufang Lee, Wei-Tek Tsai, and Yinong Chen. 2008. Ontology-based test modeling and partition testing of web services. In Proceedings of the IEEE International Conference on Web Services (ICWS’08). IEEE, 465--472. DOI:https://doi.org/10.1109/ICWS.2008.111Google ScholarDigital Library
- Xiaoying Bai, Dezheng Xu, Guilan Dai, Wei-Tek Tsai, and Yinong Chen. 2007. Dynamic reconfigurable testing of service-oriented architecture. In Proceedings of the 31st Annual International Computer Software and Applications Conference (COMPSAC’07), Vol. 1. IEEE, 368--378. DOI:https://doi.org/10.1109/COMPSAC.2007.106Google ScholarDigital Library
- Luciano Baresi and Carlo Ghezzi. 2010. The disappearing boundary between development-time and run-time. In Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research (FoSER’10). ACM, 17--22. DOI:https://doi.org/10.1145/1882362.1882367Google ScholarDigital Library
- Cesare Bartolini, Antonia Bertolino, Sebastian Elbaum, and Eda Marchetti. 2011. Bringing white-box testing to Service Oriented Architectures through a Service Oriented Approach. J. Syst. Softw. 84, 4 (Apr. 2011), 655--668. DOI:https://doi.org/10.1016/j.jss.2010.10.024Google ScholarDigital Library
- Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos engineering. IEEE Softw. 33, 3 (2016), 35--41. DOI:https://doi.org/10.1109/MS.2016.60Google ScholarDigital Library
- Jonathan Bell, Christian Murphy, and Gail Kaiser. 2015. Metamorphic runtime checking of applications without test oracles. J. Defense Softw. Engineering 28, 2 (2015), 9--13.Google Scholar
- Antonia Bertolino, Guglielmo De Angelis, Micael Gallego, Boni García, Francisco Gortázar, Francesca Lonetti, and Eda Marchetti. 2019. A systematic review on cloud testing. Comput. Surv. 52, 5, Article 93 (Sept. 2019). DOI:https://doi.org/10.1145/3331447Google Scholar
- Antonia Bertolino, Guglielmo De Angelis, Lars Frantzen, and Andrea Polini. 2007. The PLASTIC framework and tools for testing service-oriented applications. In International Summer Schools: Software Engineering (ISSSE 2006-2008), Andrea De Lucia and Filomena Ferrucci (Eds.). Lecture Notes in Computer Science, Vol. 5413. Springer, 106--139. DOI:https://doi.org/10.1007/978-3-540-95888-8_5Google Scholar
- Antonia Bertolino, Guglielmo De Angelis, Sampo Kellomaki, and Andrea Polini. 2012. Enhancing service federation trustworthiness through online testing. Computer 45, 1 (2012), 66--72. DOI:https://doi.org/10.1109/MC.2011.227Google ScholarDigital Library
- Antonia Bertolino, Guglielmo De Angelis, and Andrea Polini. 2012. Governance policies for verification and validation of service choreographies. In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST’12), J. Cordeiro and K.H. Krempels (Eds.), Lecture Notes in Business Information Processing, Vol. 140. Springer, 86--102. DOI:https://doi.org/10.1007/978-3-642-36608-6_6Google Scholar
- Antonia Bertolino and Andrea Polini. 2009. SOA test governance: Enabling service integration testing across organization and technology borders. In Proceedings of the International Workshop on Web Testing (WebTest’09). IEEE, 277--286. DOI:https://doi.org/10.1109/ICSTW.2009.39Google ScholarDigital Library
- Smt. J. Sasi Bhanu, A. Vinaya Babu, and P. Trimurthy. 2015. A comprehensive architecture for dynamic evolution of online testing of an embedded system. J. Theor. Appl. Inf. Technol. 80, 1 (Oct. 2015), 160--172. http://www.jatit.org/volumes/Vol80No1/17Vol80No1.pdfGoogle Scholar
- Jayaram Bobba, Weiwei Xiong, Luke Yen, Mark D. Hill, and David A. Wood. 2009. StealthTest: Low overhead online software testing using transactional memory. In Proceedings of the 18th International Conference on Parallel Architectures and Compilation Techniques (PACT’09). IEEE, 146--155. DOI:https://doi.org/10.1109/PACT.2009.15Google Scholar
- Daniel Brenner, Colin Atkinson, Barbara Paech, Rainer Malaka, Matthias Merdes, and Dima Suliman. 2006. Reducing verification effort in component-based software engineering through built-in testing. In Proceedings of the 10th IEEE International Enterprise Distributed Object Computing Conference (EDOC’06). IEEE, 175--184. DOI:https://doi.org/10.1109/EDOC.2006.44Google ScholarDigital Library
- Kai-Yuan Cai, Yong-Chao Li, and Ke Liu. 2004. Optimal and adaptive testing for software reliability assessment. Inf. Softw. Technol. 46, 15 (Dec. 2004), 989--1000. DOI:https://doi.org/10.1016/j.infsof.2004.07.006Google ScholarCross Ref
- Tien-Dung Cao, Patrick Félix, Richard Castanet, and Ismail Berrada. 2010. Online testing framework for web services. In Proceedings of the 3rd International Conference on Software Testing, Verification and Validation (ICST’10). IEEE, 363--372. DOI:https://doi.org/10.1109/ICST.2010.11Google ScholarDigital Library
- W. K. Chan, S. C. Cheung, and Karl R. P. H. Leung. 2007. A metamorphic testing approach for online testing of service-oriented software applications. Int. J. Web Serv. Res. 4, 2 (2007), 61--81. DOI:https://doi.org/10.4018/jwsr.2007040103Google ScholarCross Ref
- Mark B. Cooray, James H. Hamlyn-Harris, and Robert G. Merkel. 2015. Dynamic test reconfiguration for composite web services. IEEE Trans. Serv. Comput. 8, 4 (2015), 576--585. DOI:https://doi.org/10.1109/TSC.2014.2312953Google ScholarCross Ref
- Andrew Diniz da Costa, Camila Nunes, Viviane Torres da Silva, Baldoino Fonseca, and Carlos J. P. de Lucena. 2010. JAAF+T: A framework to implement self-adaptive agents that apply self-test. In Proceedings of the 2010 ACM Symposium on Applied Computing (SAC’10). ACM, 928--935. DOI:https://doi.org/10.1145/1774088.1774280Google Scholar
- Huning Dai, Christian Murphy, and Gail Kaiser. 2010. Configuration fuzzing for software vulnerability detection. In Proceedings of the 5th International Conference on Availability, Reliability and Security (ARES’10). IEEE, 525--530. DOI:https://doi.org/10.1109/ARES.2010.22Google ScholarCross Ref
- Huning Dai, Christian Murphy, and Gail E. Kaiser. 2010. CONFU: Configuration fuzzing testing framework for software vulnerability detection. Int. J. Sec. Softw. Eng. 1, 3 (2010), 41--55. DOI:https://doi.org/10.4018/jsse.2010070103Google ScholarDigital Library
- Guglielmo De Angelis, Antonia Bertolino, and Andrea Polini. 2011. (role)CAST: A framework for on-line service testing. In Proceedings of the 7th International Conference on Web Information Systems and Technologies (WEBIST’11). SciTePress, Noordwijkerhout, The Netherlands, 13--18. DOI:https://doi.org/10.5220/0003340500130018Google Scholar
- Guglielmo De Angelis, Antonia Bertolino, and Andrea Polini. 2012. Validation and verification policies for governance of service choreographies. In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST’12). SciTePress, 58--70. DOI:https://doi.org/10.5220/0003936200580070Google Scholar
- Vânia de Oliveira Neves, Márcio Eduardo Delamaro, and Paulo Cesar Masiero. 2014. An environment to support structural testing of autonomous vehicles. In Proceedings of the IV Brazilian Symposium on Computing Systems Engineering (SBESC’14). IEEE, 19--24. DOI:https://doi.org/10.1109/SBESC.2014.27Google ScholarDigital Library
- Pouria Derakhshanfar, Xavier Devroey, Gilles Perrouin, Andy Zaidman, and Arie van Deursen. 2020. Search-based crash reproduction using behavioural model seeding. Softw. Test. Verif. Reliab. 30, 3, Article e1733 (May 2020). DOI:https://doi.org/10.1002/stvr.1733Google Scholar
- Peter H. Deussen, George Din, and Ina Schieferdecker. 2002. An on-line test platform for component-based systems. In Proceedings of the 27th Annual NASA Goddard Software Engineering Workshop (SEW’02). IEEE, 96--103. DOI:https://doi.org/10.1109/SEW.2002.1199455Google ScholarDigital Library
- Peter H. Deussen, George Din, and Ina Schieferdecker. 2003. A TTCN-3 based online test and validation platform for Internet services. In Proceedings of the 6th International Symposium on Autonomous Decentralized Systems (ISADS’03). IEEE, 177--184. DOI:https://doi.org/10.1109/ISADS.2003.1193946Google ScholarDigital Library
- Massimiliano Di Penta, Marcello Bruno, Gianpiero Esposito, Valentina Mazza, and Gerardo Canfora. 2007. Web Services Regression Testing. Springer, 205--234. DOI:https://doi.org/10.1007/978-3-540-72912-9_8Google Scholar
- Dimitris Dranidis, Andreas Metzger, and Dimitrios Kourtesis. 2010. Enabling proactive adaptation through just-in-time testing of conversational services. In Proceedings of the 3rd European Conference (ServiceWave’10), Elisabetta Di Nitto and Ramin Yahyapour (Eds.), Lecture Notes in Computer Science,Vol. 6481. Springer, 63--75. DOI:https://doi.org/10.1007/978-3-642-17694-4_6Google ScholarCross Ref
- Sebastian Elbaum and Madeline Hardojo. 2004. An empirical study of profiling strategies for released software and their impact on testing activities. In Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’04). ACM, 65--75. DOI:https://doi.org/10.1145/1007512.1007522Google ScholarDigital Library
- Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering: A roadmap and agenda. J. Syst. Softw. 123, 1 (Jan. 2017), 176--189. DOI:https://doi.org/10.1016/j.jss.2015.06.063Google ScholarCross Ref
- Lars Frantzen, Jan Tretmans, and Tim A. C. Willemse. 2004. Test generation based on symbolic specifications. In Proceedings of 4th International Workshop on Formal Approaches to Software Testing (FATES’04), Jens Grabowski and Brian Nielsen (Eds.), Lecture Notes in Computer Science,Vol. 3395. Springer, 1--15. DOI:https://doi.org/10.1007/978-3-540-31848-4_1Google Scholar
- Erik M. Fredericks and Betty H. C. Cheng. 2015. Automated generation of adaptive test plans for self-adaptive systems. In Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’15). IEEE, 157--167. DOI:https://doi.org/10.1109/SEAMS.2015.15Google Scholar
- Erik M. Fredericks, Byron DeVries, and Betty H. C. Cheng. 2014. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty. In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’14). ACM, 17--26. DOI:https://doi.org/10.1145/2593929.2593937Google Scholar
- Luca Gazzola, Leonardo Mariani, Fabrizio Pastore, and Mauro Pezzè. 2017. An exploratory study of field failures. In Proceedings of the 28th International Symposium on Software Reliability Engineering (ISSRE’17). IEEE, 67--77. DOI:https://doi.org/10.1109/ISSRE.2017.10Google ScholarCross Ref
- Okehee Goh and Yann-Hang Lee. 2007. Schedulable online testing framework for real-time embedded applications in VM. In Proceedings of the International Conference on Embedded and Ubiquitous Computing (EUC’07), Tei-Wei Kuo, Edwin Sha, Minyi Guo, Laurence T. Yang, and Zili Shao (Eds.), Lecture Notes in Computer Science,Vol. 4808. Springer, 730--741. DOI:https://doi.org/10.1007/978-3-540-77092-3_63Google ScholarCross Ref
- Alberto González, Eric Piel, and Hans-Gerhard Gross. 2008. Architecture support for runtime integration and verification of component-based systems of systems. In Proceedings of the International Workshop on Automated Engineering of Autonomous and Run-tiMe Evolving Systems (ARAMIS’08). IEEE, 41--48. DOI:https://doi.org/10.1109/ASEW.2008.4686292Google ScholarDigital Library
- Alberto Gonzalez-Sanchez, Eric Piel, and Hans-Gerhard Gross. 2009. RiTMO: A method for runtime testability measurement and optimisation. In Proceedings of the 9th International Conference on Quality Software (QSIC’09). IEEE, 377--382. DOI:https://doi.org/10.1109/QSIC.2009.56Google ScholarDigital Library
- Alberto Gonzalez-Sanchez, Eric Piel, Hans-Gerhard Gross, and Arjan J. C. van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems. In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID’10). IEEE, 37--42. DOI:https://doi.org/10.1109/VALID.2010.13Google Scholar
- Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Jian Lü, and Zhendong Su. 2016. Automatic runtime recovery via error handler synthesis. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). ACM, 684--695. DOI:https://doi.org/10.1145/2970276.2970360Google ScholarDigital Library
- Petr Hosek and Cristian Cadar. 2015. VARAN the unbelievable: An efficient n-version execution framework. In Proceedings of the 20th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’15), ACM SIGPLAN Notices, Vol. 50. ACM, 339--353. DOI:https://doi.org/10.1145/2694344.2694390Google ScholarDigital Library
- Zhan-Wei Hui, Song Huang, and Meng-Yu Ji. 2016. A runtime-testing method for integer overflow detection based on metamorphic relations. J. Intell. Fuzzy Syst. 31, 4 (2016), 2349--2361. DOI:https://doi.org/10.3233/JIFS-169076Google ScholarDigital Library
- Waldemar Hummer, Orna Raz, Onn Shehory, Philipp Leitner, and Schahram Dustdar. 2011. Test coverage of data-centric dynamic compositions in service-based systems. In Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation (ICST’11). IEEE, 40--49. DOI:https://doi.org/10.1109/ICST.2011.55Google ScholarDigital Library
- Waldemar Hummer, Orna Raz, Onn Shehory, Philipp Leitner, and Schahram Dustdar. 2013. Testing of data-centric and event-based dynamic service compositions. Softw. Test. Verif. Reliabil. 23, 6 (Sep. 2013), 465--497. DOI:https://doi.org/10.1002/stvr.1493Google Scholar
- Melody Y. Ivory and Marti A. Hearst. 2001. The state of the art in automating usability evaluation of user interfaces. Comput. Surv. 33, 4 (Dec. 2001), 470--516. DOI:https://doi.org/10.1145/503112.503114Google ScholarDigital Library
- Wei Jin and Alessandro Orso. 2012. BugRedux: Reproducing field failures for in-house debugging. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE, 474--484. DOI:https://doi.org/10.1109/ICSE.2012.6227168Google ScholarDigital Library
- Katsumi Kawano, Masayuki Orimo, and Kinji Mori. 1989. Autonomous decentralized system test technique. In Proceedings of the 13th Annual International Computer Software & Applications Conference (COMPSAC’89). IEEE, 52--57. DOI:https://doi.org/10.1109/CMPSAC.1989.65053Google ScholarCross Ref
- Fitsum Meshesha Kifetew, Wei Jin, Roberto Tiella, Alessandro Orso, and Paolo Tonella. 2014. Reproducing field failures for programs with complex grammar-based input. In Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation (ICST’14). IEEE, 163--172. DOI:https://doi.org/10.1109/ICST.2014.29Google ScholarDigital Library
- Tariq M. King, Andrew A. Allen, Rodolfo Cruz, and Peter J. Clarke. 2011. Safe runtime validation of behavioral adaptations in autonomic software. In Proceedings of the 8th International Conference on Autonomic and Trusted Computing (ATC’11), Jose M. Alcaraz Calero, Laurence T. Yang, Félix Gómez Mármol, and Luis Javier García Villalba (Eds.), Lecture Notes in Computer Science, Vol. 6906. Springer, 31--46. DOI:https://doi.org/10.1007/978-3-642-23496-5_3Google Scholar
- Tariq M. King, Djuradj Babich, Jonatan Alava, Peter J. Clarke, and Ronald Stevens. 2007. Towards self-testing in autonomic computing systems. In Proceedings of the 8th International Symposium on Autonomous Decentralized Systems (ISADS’07). IEEE, 51--58. DOI:https://doi.org/10.1109/ISADS.2007.75Google ScholarDigital Library
- Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, and Peter J. Clarke. 2007. An integrated self-testing framework for autonomic computing systems. J. Comput. 2, 9 (2007), 37--49. DOI:https://doi.org/10.4304/jcp.2.9.37-49Google ScholarCross Ref
- Barbara Kitchenham and Stuart Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE-2007-01.Google Scholar
- Ron Kohavi and Roger Longbotham. 2017. Online controlled experiments and A/B testing. In Encyclopedia of Machine Learning and Data Mining, Claude Sammut and Geoffrey I. Webb (Eds.). Springer, 922--929. DOI:https://doi.org/10.1007/978-1-4899-7687-1_891Google Scholar
- Mariam Lahami, Fairouz Fakhfakh, Moez Krichen, and Mohamed Jmaiel. 2012. Towards a TTCN-3 test system for runtime testing of adaptable and distributed systems. In Proceedings of the 24th IFIP WG 6.1 International Conference on Testing Software and Systems (ICTSS’12), Brian Nielsen and Carsten Weise (Eds.), Lecture Notes in Computer Science,Vol. 7641. Springer, 71--86. DOI:https://doi.org/10.1007/978-3-642-34691-0_7Google ScholarCross Ref
- Mariam Lahami and Moez Krichen. 2013. Test isolation policy for safe runtime validation of evolvable software systems. In Proceedings of the Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE’13). IEEE, 377--382. DOI:https://doi.org/10.1109/WETICE.2013.62Google ScholarDigital Library
- Mariam Lahami, Moez Krichen, Mariam Bouchakwa, and Mohamed Jmaiel. 2012. Using knapsack problem model to design a resource aware test architecture for adaptable and distributed systems. In Proceedings of the 24th IFIP WG 6.1 International Conference on Testing Software and Systems (ICTSS’12), Brian Nielsen and Carsten Weise (Eds.), Lecture Notes in Computer Science,Vol. 7641. Springer, 103--118. DOI:https://doi.org/10.1007/978-3-642-34691-0_9Google ScholarCross Ref
- Mariam Lahami, Moez Krichen, and Mohamed Jmaiel. 2012. A distributed test architecture for adaptable and distributed real-time systems. In Avancées Récentes Dans le Domaine Des Architectures Logicielles: Articles Sélectionnés et Etendus de (CAL'11) (Revue des Nouvelles Technologies de l'Information), Philippe Aniorté (Ed.), Vol. L-6. Hermann, 732--92.Google Scholar
- Mariam Lahami, Moez Krichen, and Mohamed Jmaiel. 2013. Runtime testing framework for improving quality in dynamic service-based systems. In Proceedings of the 2nd International Workshop on Quality Assurance for Service-based Applications (QASBA’13). ACM, 17--24. DOI:https://doi.org/10.1145/2489300.2489335Google ScholarDigital Library
- Mariam Lahami, Moez Krichen, and Mohamed Jmaïel. 2015. Runtime testing approach of structural adaptations for dynamic and distributed systems. Int. J. Comput. Appl. Technol. 51, 4 (Jul. 2015), 259--272. DOI:https://doi.org/10.1504/IJCAT.2015.070489Google ScholarDigital Library
- Mariam Lahami, Moez Krichen, and Mohamed Jmaiel. 2016. Safe and efficient runtime testing framework applied in dynamic and distributed systems. Sci. Comput. Program. 122, 1 (2016), 1--28. DOI:https://doi.org/10.1016/j.scico.2016.02.002Google ScholarDigital Library
- Gregory L. Lee, Dong H. Ahn, Dorian C. Arnold, Bronis R. de Supinski, Matthew Legendre, Barton P. Miller, Martin Schulz, and Ben Liblit. 2008. Lessons learned at 208K: Towards debugging millions of cores. In Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (SC’08). IEEE, 1--9. DOI:https://doi.org/10.1109/SC.2008.5218557Google ScholarCross Ref
- Kihwal Lee and Lui Sha. 2005. A dependable online testing and upgrade architecture for real-time embedded systems. In Proceedings of the 11th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA’05). IEEE, 160--165. DOI:https://doi.org/10.1109/RTCSA.2005.8Google ScholarDigital Library
- Seoung-Hyeon Lee and Jung-Chan Na. 2017. Development of test agents for automated dynamic testing of UNIWAY. In Proceedings of the International Conference on Advanced Multimedia and Ubiquitous Engineering (MUE/FutureTech’17), James J. (Jong Hyuk) Park, Shu-Ching Chen, and Kim-Kwang Raymond Choo (Eds.), Lecture Notes in Electrical Engineering, Vol. 448. Springer, 683--688. DOI:https://doi.org/10.1007/978-981-10-5041-1_109Google ScholarCross Ref
- Martin Leucker and Christian Schallhart. 2009. A brief account of runtime verification. J. Logic Algebr. Program. 78, 5 (2009), 293--303. DOI:https://doi.org/10.1016/j.jlap.2008.08.004Google ScholarCross Ref
- Riku Luostarinen, Risto Järvinen, Juho Määttä, and Jukka Manner. 2016. A model for usability testing in challenging environments. In Proceedings of the International Conference on Military Communications and Information Systems (ICMCIS’16). IEEE, 1--6. DOI:https://doi.org/10.1109/ICMCIS.2016.7496557Google ScholarCross Ref
- Bo Ma, Bin Chen, Xiaoying Bai, and Junfei Huang. 2010. Design of BDI agent for adaptive performance testing of web services. In Proceedings of the 10th International Conference on Quality Software (QSIC’10). IEEE, 435--440. DOI:https://doi.org/10.1109/QSIC.2010.69Google ScholarDigital Library
- Afef Jmal Maâlej, Moez Krichen, and Mohamed Jmaïel. 2012. Model-based conformance testing of WS-BPEL compositions. In Proceedings of the 4th IEEE International Workshop on Software Testing Automation (STA’12). IEEE, 452--457. DOI:https://doi.org/10.1109/COMPSACW.2012.86Google ScholarDigital Library
- Lijun Mei, W. K. Chan, T. H. Tse, Bo Jiang, and Ke Zhai. 2015. Preemptive regression testing of workflow-based web services. IEEE Trans. Serv. Comput. 8, 5 (Sep. 2015), 740--754. DOI:https://doi.org/10.1109/TSC.2014.2322621Google ScholarCross Ref
- Andreas Metzger, Osama Sammodi, Klaus Pohl, and Mark Rzepka. 2010. Towards pro-active adaptation with confidence: Augmenting service monitoring with online testing. In Proceedings of the 2010 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’10). ACM, 20--28. DOI:https://doi.org/10.1145/1808984.1808987Google ScholarDigital Library
- Jesús Morán, Antonia Bertolino, Claudio de la Riva, and Javier Tuya. 2017. Towards ex Vivo testing of mapreduce applications. In Proceedings of the 2017 International Conference on Software Quality, Reliability and Security (QRS’17). IEEE, 73--80. DOI:https://doi.org/10.1109/QRS.2017.17Google Scholar
- Christian Murphy, Gail E. Kaiser, Ian Vo, and Matt Chu. 2009. Quality assurance of software applications using the in Vivo testing approach. In Proceedings of the 2nd International Conference on Software Testing Verification and Validation (ICST’09). IEEE, 111--120. DOI:https://doi.org/10.1109/ICST.2009.18Google ScholarDigital Library
- Christian Murphy, Kuang Shen, and Gail Kaiser. 2009. Automatic system testing of programs without test oracles. In Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA’09). ACM, 189--200. DOI:https://doi.org/10.1145/1572272.1572295Google ScholarDigital Library
- Christian Murphy, Moses Vaughan, Waseem Ilahi, and Gail Kaiser. 2010. Automatic detection of previously-unseen application states for deployment environment testing and analysis. In Proceedings of the 5th Workshop on Automation of Software Test (AST’10). ACM, 16--23. DOI:https://doi.org/10.1145/1808266.1808269Google ScholarDigital Library
- Vânia de Oliveira Neves, Márcio Eduardo Delamaro, and Paulo Cesar Masiero. 2016. Combination and mutation strategies to support test data generation in the context of autonomous vehicles. Int. J. Embed. Syst. 8, 5/6 (2016), 464--482. DOI:https://doi.org/10.1504/IJES.2016.080388Google ScholarCross Ref
- Vânia de Oliveira Neves, Márcio Eduardo Delamaro, and Paulo Cesar Masiero. 2017. Automated structural software testing of autonomous vehicles. In Proceedings of the 20th Iberoamerican Conference on Software Engineering (CibSE’17), Vol. 1. Curran Associates, Buenos Aires, Argentina, 15--28.Google Scholar
- Dirk Niebuhr, Andreas Rausch, Cornel Klein, Juergen Reichmann, and Reiner N. Schmid. 2009. Achieving dependable component bindings in dynamic adaptive systems - A runtime testing approach. In Proceedings of the 3rd IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO’09). IEEE, 186--197. DOI:https://doi.org/10.1109/SASO.2009.40Google Scholar
- Eiji Nishijima, Hiroshi Yamamoto, Katsumi Kawano, Kazunori Fujiwara, and Keiji Oshima. 1996. On-line testing for application software of widely distributed system. In Proceedings of the 15th Symposium on Reliable Distributed Systems (SRDS’96). IEEE, 54--63. DOI:https://doi.org/10.1109/RELDIS.1996.559698Google ScholarCross Ref
- Marc Oriol, Xavier Franch, and Jordi Marco. 2015. Monitoring the service-based system lifecycle with SALMon. Expert Syst. Appl. 42, 19 (Nov. 2015), 6507--6521. DOI:https://doi.org/10.1016/j.eswa.2015.03.027Google ScholarDigital Library
- Youngki Park, Woosung Jung, Byungjeong Lee, and Chisu Wu. 2009. Automatic discovery of web services based on dynamic black-box testing. In Proceedings of the 33rd Annual IEEE International Computer Software and Applications Conference (COMPSAC’09), Vol. 1. IEEE, 107--114. DOI:https://doi.org/10.1109/COMPSAC.2009.24Google ScholarDigital Library
- Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Inf. Softw. Technol. 64 (Aug. 2015), 1--18. DOI:https://doi.org/10.1016/j.infsof.2015.03.007Google ScholarDigital Library
- Mauro Pezzè and Michal Young. 2008. Software Testing and Analysis: Process, Principles, and Techniques. John Wiley & Sons.Google Scholar
- Éric Piel and Alberto Gonzalez-Sanchez. 2009. Data-flow integration testing adapted to runtime evolution in component-based systems. In Proceedings of the 2009 ESEC/FSE workshop on Software Integration and Evolution @ Runtime (SINTER’09). ACM, Amsterdam, The Netherlands, 3--10. DOI:https://doi.org/10.1145/1596495.1596499Google ScholarDigital Library
- Adam Porter, Cemal Yilmaz, Atif M. Memon, Douglas C. Schmidt, and Bala Natarajan. 2007. Skoll: A process and infrastructure for distributed continuous quality assurance. IEEE Trans. Softw. Eng. 33, 8 (2007), 510--525. DOI:https://doi.org/10.1109/TSE.2007.70719Google ScholarDigital Library
- Jeremias Rößler, Andreas Zeller, Gordon Fraser, Cristian Zamfir, and George Candea. 2013. Reconstructing core dumps. In Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation, (ICST’13). IEEE, 114--123. DOI:https://doi.org/10.1109/ICST.2013.18Google ScholarDigital Library
- Osama Sammodi, Andreas Metzger, Xavier Franch, Marc Oriol, Jordi Marco, and Klaus Pohl. 2011. Usage-based online testing for proactive adaptation of service-based applications. In Proceedings of the 35th Annual IEEE International Computer Software and Applications Conference (COMPSAC’11). IEEE, 582--587. DOI:https://doi.org/10.1109/COMPSAC.2011.81Google ScholarDigital Library
- Gerald Schermann, Jürgen Cito, and Philipp Leitner. 2018. Continuous experimentation: Challenges, implementation techniques, and current research. IEEE Softw. 35, 2 (2018), 26--31. DOI:https://doi.org/10.1109/MS.2018.111094748Google ScholarCross Ref
- Carol Smidts, Chetan Mutha, Manuel Rodriguez, and Matthew J. Gerber. 2014. Software testing with an operational profile: OP definition. Comput. Surv. 46, 3, Article 39 (Feb. 2014). DOI:https://doi.org/10.1145/2518106Google Scholar
- Mozhan Soltani, Pouria Derakhshanfar, Xavier Devroey, and Arie van Deursen. 2020. A benchmark-based evaluation of search-based crash reproduction. Empir. Softw. Eng. 25, 1 (Jan. 2020), 96--138. DOI:https://doi.org/10.1007/s10664-019-09762-1Google ScholarDigital Library
- Ronald Stevens, Brittany Parsons, and Tariq M. King. 2007. A self-testing autonomic container. In Proceedings of the 45th Annual Southeast Regional Conference (ACM-SE 45). ACM, 1--6. DOI:https://doi.org/10.1145/1233341.1233343Google Scholar
- Dima Suliman, Barbara Paech, Lars Borner, Colin Atkinson, Daniel Brenner, Matthias Merdes, and Rainer Malaka. 2006. The MORABIT approach to runtime component testing. In Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC’06), Vol. 2. IEEE, 171--176. DOI:https://doi.org/10.1109/COMPSAC.2006.169Google ScholarDigital Library
- Jüri Vain, Leonidas Tsiopoulos, Vyacheslav Kharchenko, Apneet Kaur, Maksim Jenihhin, and Jaan Raik. 2017. Multi-fragment Markov model guided online test generation for MPSoC. In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT’17). 594--607.Google Scholar
- Margus Veanes, Colin Campbell, Wolfram Schulte, and Nikolai Tillmann. 2005. Online testing with model programs. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13), ACM SIGSOFT Software Engineering Notes,Vol. 30. ACM, 273--282. DOI:https://doi.org/10.1145/1081706.1081751Google ScholarDigital Library
- Arnold P. O. S. Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Väänänen-Vainio-Mattila. 2010. User experience evaluation methods: Current state and development needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI’10). ACM, 521--530. DOI:https://doi.org/10.1145/1868914.1868973Google ScholarDigital Library
- Yongbo Wang, Xiaoying Bai, Juanzi Li, and Ruobo Huang. 2007. Ontology-based test case generation for testing web services. In Proceedings of the 8th International Symposium on Autonomous Decentralized Systems (ISADS’07). IEEE, 43--50. DOI:https://doi.org/10.1109/ISADS.2007.54Google ScholarDigital Library
- Yingxu Wang, Graham King, Dilip Patel, Shushma Patel, and Alec Dorling. 1999. On coping with real-time software dynamic inconsistency by built-in tests. Ann. Softw. Eng. 7, 1 (Oct. 1999), 283--296. DOI:https://doi.org/10.1023/A:1018990322378Google ScholarDigital Library
- Hiroshi Yamamoto, Akira Yoshizawa, Katsumi Kawano, Kinji Mori, Keiji Oshima, and Tsunao Kawamura. 1993. On-line software test techniques based on autonomous decentralized system. In Proceedings of the 4th Workshop on Future Trends of Distributed Computing Systems (FTDCS’93). IEEE, 291--296. DOI:https://doi.org/10.1109/FTDCS.1993.344143Google ScholarCross Ref
- Chunyang Ye and Hans-Arno Jacobsen. 2013. Whitening SOA testing via event exposure. IEEE Trans. Softw. Eng. 39, 10 (2013), 1444--1465. DOI:https://doi.org/10.1109/TSE.2013.20Google ScholarDigital Library
- Jia Zhang. 2004. An approach to facilitate reliability testing of web services components. In Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). IEEE, 210--218. DOI:https://doi.org/10.1109/ISSRE.2004.4Google ScholarDigital Library
- Hong Zhu. 2006. A framework for service-oriented testing of web services. In Proceedings of the 3rd International Workshop on Quality Assurance and Testing Web-Based Applications (QATWBA’06), Vol. 2. IEEE, 145--150. DOI:https://doi.org/10.1109/COMPSAC.2006.95Google ScholarDigital Library
- Hong Zhu and Yufeng Zhang. 2012. Collaborative testing of web services. IEEE Trans. Serv. Comput. 5, 1 (2012), 116--130. DOI:https://doi.org/10.1109/TSC.2010.54Google ScholarDigital Library
- Hong Zhu and Yufeng Zhang. 2014. A test automation framework for collaborative testing of web service dynamic compositions. In Advanced Web Services, Athman Bouguettaya, Quan Z. Sheng, and Florian Daniel (Eds.). Springer, 171--197. DOI:https://doi.org/10.1007/978-1-4614-7535-4_8Google Scholar
Index Terms
- A Survey of Field-based Testing Techniques
Recommendations
Flakiness goes live: Insights from an In Vivo testing simulation study
Abstract Context:Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as flaky are put aside for being either repaired or discarded.
Objective:We hypothesise that some ...
Fault-based test suite prioritization for specification-based testing
Context: Existing test suite prioritization techniques usually rely on code coverage information or historical execution data that serve as indicators for estimating the fault-detecting ability of test cases. Such indicators are primarily empirical in ...
The experimental applications of search-based techniques for model-based testing
Graphical abstractDisplay Omitted HighlightsA systematic review of applications of search-based techniques for model-based testing is provided.Four taxonomies are proposed to classify the applications based on the purpose, problems, solutions and ...
Comments