Skip to main content
Erschienen in: Automated Software Engineering 1/2019

15.10.2018

Exploring output-based coverage for testing PHP web applications

verfasst von: Hung Viet Nguyen, Hung Dang Phan, Christian Kästner, Tien N. Nguyen

Erschienen in: Automated Software Engineering | Ausgabe 1/2019

Einloggen

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

In software testing, different testers focus on different aspects of the software such as functionality, performance, design, and other attributes. While many tools and coverage metrics exist to support testers at the code level, not much support is targeted for testers who want to inspect the output of a program such as a dynamic web application. To support this category of testers, we propose a family of output-coverage metrics (similar to statement, branch, and path coverage metrics on code) that measure how much of the possible output has been produced by a test suite and what parts of the output are still uncovered. To do that, we first approximate the output universe using our existing symbolic execution technique. Then, given a set of test cases, we map the produced outputs onto the output universe to identify the covered and uncovered parts and compute output-coverage metrics. In our empirical evaluation on seven real-world PHP web applications, we show that selecting test cases by output coverage is more effective at identifying presentation faults such as HTML validation errors and spelling errors than selecting test cases by traditional code coverage. In addition, to help testers understand output coverage and augment test cases, we also develop a tool called WebTest that displays the output universe in one single web page and allows testers to visually explore covered and uncovered parts of the output.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
Zurück zum Zitat Alshahwan, N., Harman, M.: Automated web application testing using search based software engineering. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE ’11, Washington, DC, USA, pp. 3–12. IEEE Computer Society (2011). https://doi.org/10.1109/ASE.2011.6100082. ISBN 978-1-4577-1638-6 Alshahwan, N., Harman, M.: Automated web application testing using search based software engineering. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE ’11, Washington, DC, USA, pp. 3–12. IEEE Computer Society (2011). https://​doi.​org/​10.​1109/​ASE.​2011.​6100082. ISBN 978-1-4577-1638-6
Zurück zum Zitat Alshahwan, N., Harman, M.: Augmenting test suites effectiveness by increasing output diversity. In: Proceedings of the 34th International Conference on Software Engineering, ICSE ’12, pp. 1345–1348. IEEE Press (2012) Alshahwan, N., Harman, M.: Augmenting test suites effectiveness by increasing output diversity. In: Proceedings of the 34th International Conference on Software Engineering, ICSE ’12, pp. 1345–1348. IEEE Press (2012)
Zurück zum Zitat Alshahwan, N., Harman, M.: Coverage and fault detection of the output-uniqueness test selection criteria. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, New York, NY, USA, pp. 181–192. ACM (2014) Alshahwan, N., Harman, M.: Coverage and fault detection of the output-uniqueness test selection criteria. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, New York, NY, USA, pp. 181–192. ACM (2014)
Zurück zum Zitat Ammann, P., Offutt, J.: Introduction to Software Testing, 1st edn. Cambridge University Press, New York (2008)CrossRef Ammann, P., Offutt, J.: Introduction to Software Testing, 1st edn. Cambridge University Press, New York (2008)CrossRef
Zurück zum Zitat Andrews, A.A., Offutt, J., Alexander, R.T.: Testing web applications by modeling with FSMs. Softw. Syst. Model. 4, 326–345 (2005)CrossRef Andrews, A.A., Offutt, J., Alexander, R.T.: Testing web applications by modeling with FSMs. Softw. Syst. Model. 4, 326–345 (2005)CrossRef
Zurück zum Zitat Artzi, S., Dolby, J., Jensen, S.H., Møller, A., Tip, F.: A framework for automated testing of JavaScript web applications. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pp. 571–580, New York, NY, USA. ACM (2011). https://doi.org/10.1145/1985793.1985871. ISBN 978-1-4503-0445-0 Artzi, S., Dolby, J., Jensen, S.H., Møller, A., Tip, F.: A framework for automated testing of JavaScript web applications. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pp. 571–580, New York, NY, USA. ACM (2011). https://​doi.​org/​10.​1145/​1985793.​1985871. ISBN 978-1-4503-0445-0
Zurück zum Zitat Elbaum, S., Karre, S., Rothermel, G.: Improving web application testing with user session data. In: Proceedings of the 25th International Conference on Software Engineering, ICSE ’03, Washington, DC, USA, pp. 49–59. IEEE Computer Society (2003). http://dl.acm.org/citation.cfm?id=776816.776823. ISBN 0-7695-1877-X. Accessed 30 June 2017 Elbaum, S., Karre, S., Rothermel, G.: Improving web application testing with user session data. In: Proceedings of the 25th International Conference on Software Engineering, ICSE ’03, Washington, DC, USA, pp. 49–59. IEEE Computer Society (2003). http://​dl.​acm.​org/​citation.​cfm?​id=​776816.​776823. ISBN 0-7695-1877-X. Accessed 30 June 2017
Zurück zum Zitat Elbaum, S., Chilakamarri, K.-R., Fisher II, M., Rothermel, G.: Web application characterization through directed requests. In: Proceedings of the 2006 International Workshop on Dynamic Systems Analysis, WODA ’06, New York, NY, USA, pp. 49–56. ACM (2006). https://doi.org/10.1145/1138912.1138923. ISBN 1-59593-400-6 Elbaum, S., Chilakamarri, K.-R., Fisher II, M., Rothermel, G.: Web application characterization through directed requests. In: Proceedings of the 2006 International Workshop on Dynamic Systems Analysis, WODA ’06, New York, NY, USA, pp. 49–56. ACM (2006). https://​doi.​org/​10.​1145/​1138912.​1138923. ISBN 1-59593-400-6
Zurück zum Zitat Girardi, C., Ricca, F., Tonella, P.: Web crawlers compared. Int. J. Web Inf. Syst. 2(2), 85–94 (2006)CrossRef Girardi, C., Ricca, F., Tonella, P.: Web crawlers compared. Int. J. Web Inf. Syst. 2(2), 85–94 (2006)CrossRef
Zurück zum Zitat Goodenough, J.B., Gerhart, S.L.: Toward a theory of test data selection. In: Proceedings of the International Conference on Reliable Software, pp. 493–510. ACM (1975) Goodenough, J.B., Gerhart, S.L.: Toward a theory of test data selection. In: Proceedings of the International Conference on Reliable Software, pp. 493–510. ACM (1975)
Zurück zum Zitat Heidegger, P., Thiemann, P.: Contract-driven testing of JavaScript code. In: Proceedings of the 48th International Conference on Objects, Models, Components, Patterns, TOOLS ’10, pp. 154–172. Springer, Berlin (2010). http://dl.acm.org/citation.cfm?id=1894386.1894395. ISBN 3-642-13952-3, 978-3-642-13952-9. Accessed 30 June 2017 Heidegger, P., Thiemann, P.: Contract-driven testing of JavaScript code. In: Proceedings of the 48th International Conference on Objects, Models, Components, Patterns, TOOLS ’10, pp. 154–172. Springer, Berlin (2010). http://​dl.​acm.​org/​citation.​cfm?​id=​1894386.​1894395. ISBN 3-642-13952-3, 978-3-642-13952-9. Accessed 30 June 2017
Zurück zum Zitat Kästner, C., Giarrusso, P.G., Rendel, T., Erdweg, S., Ostermann, K., Berger, T.: Variability-aware parsing in the presence of lexical macros and conditional compilation. In: Proceedings of the 2011 ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA ’11, New York, NY, USA, pp. 805–824. ACM (2011). https://doi.org/10.1145/2048066.2048128. ISBN 978-1-4503-0940-0 Kästner, C., Giarrusso, P.G., Rendel, T., Erdweg, S., Ostermann, K., Berger, T.: Variability-aware parsing in the presence of lexical macros and conditional compilation. In: Proceedings of the 2011 ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA ’11, New York, NY, USA, pp. 805–824. ACM (2011). https://​doi.​org/​10.​1145/​2048066.​2048128. ISBN 978-1-4503-0940-0
Zurück zum Zitat Mesbah, A., van Deursen, A.: Invariant-based automatic testing of ajax user interfaces. In: Proceedings of the 31st International Conference on Software Engineering, ICSE ’09, Washington, DC, USA, pp. 210–220. IEEE Computer Society (2009). https://doi.org/10.1109/ICSE.2009.5070522. ISBN 978-1-4244-3453-4 Mesbah, A., van Deursen, A.: Invariant-based automatic testing of ajax user interfaces. In: Proceedings of the 31st International Conference on Software Engineering, ICSE ’09, Washington, DC, USA, pp. 210–220. IEEE Computer Society (2009). https://​doi.​org/​10.​1109/​ICSE.​2009.​5070522. ISBN 978-1-4244-3453-4
Zurück zum Zitat Milani Fard, A., Mirzaaghaei, M., Mesbah, A.: Leveraging existing tests in automated test generation for web applications. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, New York, NY, USA, pp. 67–78. ACM (2014). https://doi.org/10.1145/2642937.2642991. ISBN 978-1-4503-3013-8 Milani Fard, A., Mirzaaghaei, M., Mesbah, A.: Leveraging existing tests in automated test generation for web applications. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, New York, NY, USA, pp. 67–78. ACM (2014). https://​doi.​org/​10.​1145/​2642937.​2642991. ISBN 978-1-4503-3013-8
Zurück zum Zitat Miller, J.C., Maloney, C.J.: Systematic mistake analysis of digital computer programs. Commun. ACM 6(2), 58–63 (1963)CrossRefMATH Miller, J.C., Maloney, C.J.: Systematic mistake analysis of digital computer programs. Commun. ACM 6(2), 58–63 (1963)CrossRefMATH
Zurück zum Zitat Mirzaaghaei, M., Mesbah, A.: Dom-based test adequacy criteria for web applications. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, New York, NY, USA, pp. 71–81. ACM (2014). https://doi.org/10.1145/2610384.2610406. ISBN 978-1-4503-2645-2 Mirzaaghaei, M., Mesbah, A.: Dom-based test adequacy criteria for web applications. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, New York, NY, USA, pp. 71–81. ACM (2014). https://​doi.​org/​10.​1145/​2610384.​2610406. ISBN 978-1-4503-2645-2
Zurück zum Zitat Nguyen, H.V., Nguyen, H.A., Nguyen, T.T., Nguyen, T.N.: Auto-locating and fix-propagating for HTML validation errors to PHP server-side code. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, Washington, DC, USA, pp. 13–22. IEEE Computer Society (2011) Nguyen, H.V., Nguyen, H.A., Nguyen, T.T., Nguyen, T.N.: Auto-locating and fix-propagating for HTML validation errors to PHP server-side code. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, Washington, DC, USA, pp. 13–22. IEEE Computer Society (2011)
Zurück zum Zitat Nguyen, H.V., Kästner, C., Nguyen, T.N.: Building call graphs for embedded client-side code in dynamic web applications. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, New York, NY, USA, pp. 518–529. ACM (2014) Nguyen, H.V., Kästner, C., Nguyen, T.N.: Building call graphs for embedded client-side code in dynamic web applications. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, New York, NY, USA, pp. 518–529. ACM (2014)
Zurück zum Zitat Nguyen, H.V., Kästner, C., Nguyen, T.N.: Cross-language program slicing for dynamic web applications. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, New York, NY, USA, pp. 369–380. ACM (2015a) Nguyen, H.V., Kästner, C., Nguyen, T.N.: Cross-language program slicing for dynamic web applications. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, New York, NY, USA, pp. 369–380. ACM (2015a)
Zurück zum Zitat Nguyen, H.V., Kästner, C., Nguyen, T.N.: Varis: IDE support for embedded client code in PHP web applications. In: Proceedings of the 37th International Conference on Software Engineering, vol. 2, pp. 693–696, Piscataway, NJ, USA. IEEE Press (2015b) Nguyen, H.V., Kästner, C., Nguyen, T.N.: Varis: IDE support for embedded client code in PHP web applications. In: Proceedings of the 37th International Conference on Software Engineering, vol. 2, pp. 693–696, Piscataway, NJ, USA. IEEE Press (2015b)
Zurück zum Zitat Praphamontripong, U, Offutt, J.: Applying mutation testing to web applications. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, ICSTW ’10, Washington, DC, USA, pp. 132–141. IEEE Computer Society (2010). https://doi.org/10.1109/ICSTW.2010.38. ISBN 978-0-7695-4050-4 Praphamontripong, U, Offutt, J.: Applying mutation testing to web applications. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, ICSTW ’10, Washington, DC, USA, pp. 132–141. IEEE Computer Society (2010). https://​doi.​org/​10.​1109/​ICSTW.​2010.​38. ISBN 978-0-7695-4050-4
Zurück zum Zitat Richardson, D.J., Clarke, L.A.: A partition analysis method to increase program reliability. In: Proceedings of the 5th International Conference on Software Engineering, ICSE ’81, Piscataway, NJ, USA, pp. 244–253. IEEE Press (1981). http://dl.acm.org/citation.cfm?id=800078.802537. ISBN 0-89791-146-6. Accessed 30 June 2017 Richardson, D.J., Clarke, L.A.: A partition analysis method to increase program reliability. In: Proceedings of the 5th International Conference on Software Engineering, ICSE ’81, Piscataway, NJ, USA, pp. 244–253. IEEE Press (1981). http://​dl.​acm.​org/​citation.​cfm?​id=​800078.​802537. ISBN 0-89791-146-6. Accessed 30 June 2017
Zurück zum Zitat Samimi, H., Schäfer, M., Artzi, S., Millstein, T., Tip, F., Hendren, L.: Automated repair of HTML generation errors in PHP applications using string constraint solving. In: Proceedings of the 34th International Conference on Software Engineering, Piscataway, NJ, USA, pp. 277–287. IEEE Press (2012a) Samimi, H., Schäfer, M., Artzi, S., Millstein, T., Tip, F., Hendren, L.: Automated repair of HTML generation errors in PHP applications using string constraint solving. In: Proceedings of the 34th International Conference on Software Engineering, Piscataway, NJ, USA, pp. 277–287. IEEE Press (2012a)
Zurück zum Zitat Samimi, H., Schäfer, M., Artzi, S., Millstein, T., Tip, F., Hendren, L.: Automated repair of html generation errors in PHP applications using string constraint solving. In: Proceedings of the 34th International Conference on Software Engineering, ICSE ’12, Piscataway, NJ, USA, pp. 277–287. IEEE Press (2012b). http://dl.acm.org/citation.cfm?id=2337223.2337257. ISBN 978-1-4673-1067-3. Accessed 30 June 2017 Samimi, H., Schäfer, M., Artzi, S., Millstein, T., Tip, F., Hendren, L.: Automated repair of html generation errors in PHP applications using string constraint solving. In: Proceedings of the 34th International Conference on Software Engineering, ICSE ’12, Piscataway, NJ, USA, pp. 277–287. IEEE Press (2012b). http://​dl.​acm.​org/​citation.​cfm?​id=​2337223.​2337257. ISBN 978-1-4673-1067-3. Accessed 30 June 2017
Zurück zum Zitat Saxena, P., Akhawe, D., Hanna, S., Mao, F., McCamant, S., Song, D.: A symbolic execution framework for JavaScript. In: Proceedings of the 2010 IEEE Symposium on Security and Privacy, SP ’10, Washington, DC, USA, pp. 513–528. IEEE Computer Society (2010). https://doi.org/10.1109/SP.2010.38. ISBN 978-0-7695-4035-1 Saxena, P., Akhawe, D., Hanna, S., Mao, F., McCamant, S., Song, D.: A symbolic execution framework for JavaScript. In: Proceedings of the 2010 IEEE Symposium on Security and Privacy, SP ’10, Washington, DC, USA, pp. 513–528. IEEE Computer Society (2010). https://​doi.​org/​10.​1109/​SP.​2010.​38. ISBN 978-0-7695-4035-1
Zurück zum Zitat Wang, X., Zhang, L., Xie, T., Xiong, Y., Mei, H.: Automating presentation changes in dynamic web applications via collaborative hybrid analysis. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE ’12, New York, NY, USA, pp. 16:1–16:11. ACM (2012). https://doi.org/10.1145/2393596.2393614. ISBN 978-1-4503-1614-9 Wang, X., Zhang, L., Xie, T., Xiong, Y., Mei, H.: Automating presentation changes in dynamic web applications via collaborative hybrid analysis. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE ’12, New York, NY, USA, pp. 16:1–16:11. ACM (2012). https://​doi.​org/​10.​1145/​2393596.​2393614. ISBN 978-1-4503-1614-9
Zurück zum Zitat Wassermann, G., Su, Z.: Static detection of cross-site scripting vulnerabilities. In: Proceedings of the 30th International Conference on Software Engineering, ICSE ’08, New York, NY, USA, pp. 171–180. ACM (2008). https://doi.org/10.1145/1368088.1368112. ISBN 978-1-60558-079-1 Wassermann, G., Su, Z.: Static detection of cross-site scripting vulnerabilities. In: Proceedings of the 30th International Conference on Software Engineering, ICSE ’08, New York, NY, USA, pp. 171–180. ACM (2008). https://​doi.​org/​10.​1145/​1368088.​1368112. ISBN 978-1-60558-079-1
Zurück zum Zitat Wassermann, G., Yu, D., Chander, A., Dhurjati, D., Inamura, H., Su, Z.: Dynamic test input generation for web applications. In: Proceedings of the 2008 International Symposium on Software Testing and Analysis, ISSTA ’08, New York, NY, USA, pp. 249–260. ACM (2008). https://doi.org/10.1145/1390630.1390661. ISBN 978-1-60558-050-0 Wassermann, G., Yu, D., Chander, A., Dhurjati, D., Inamura, H., Su, Z.: Dynamic test input generation for web applications. In: Proceedings of the 2008 International Symposium on Software Testing and Analysis, ISSTA ’08, New York, NY, USA, pp. 249–260. ACM (2008). https://​doi.​org/​10.​1145/​1390630.​1390661. ISBN 978-1-60558-050-0
Zurück zum Zitat Yu, F., Alkhalaf, M., Bultan, T.: Patching vulnerabilities with sanitization synthesis. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, New York, NY, USA, pp. 251–260. ACM (2011). https://doi.org/10.1145/1985793.1985828. ISBN 978-1-4503-0445-0 Yu, F., Alkhalaf, M., Bultan, T.: Patching vulnerabilities with sanitization synthesis. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, New York, NY, USA, pp. 251–260. ACM (2011). https://​doi.​org/​10.​1145/​1985793.​1985828. ISBN 978-1-4503-0445-0
Zurück zum Zitat Zou, Y., Chen, Z., Zheng, Y., Zhang, X., Gao, Z.: Virtual dom coverage for effective testing of dynamic web applications. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, New York, NY, USA, pp. 60–70. ACM (2014). https://doi.org/10.1145/2610384.2610399. ISBN 978-1-4503-2645-2 Zou, Y., Chen, Z., Zheng, Y., Zhang, X., Gao, Z.: Virtual dom coverage for effective testing of dynamic web applications. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, New York, NY, USA, pp. 60–70. ACM (2014). https://​doi.​org/​10.​1145/​2610384.​2610399. ISBN 978-1-4503-2645-2
Metadaten
Titel
Exploring output-based coverage for testing PHP web applications
verfasst von
Hung Viet Nguyen
Hung Dang Phan
Christian Kästner
Tien N. Nguyen
Publikationsdatum
15.10.2018
Verlag
Springer US
Erschienen in
Automated Software Engineering / Ausgabe 1/2019
Print ISSN: 0928-8910
Elektronische ISSN: 1573-7535
DOI
https://doi.org/10.1007/s10515-018-0246-5

Weitere Artikel der Ausgabe 1/2019

Automated Software Engineering 1/2019 Zur Ausgabe