Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Software and Systems Modeling
Published in: Software and Systems Modeling 1/2017

19-04-2015 | Theme Section Paper

Variability testing in the wild: the Drupal case study

Authors: Ana B. Sánchez, Sergio Segura, José A. Parejo, Antonio Ruiz-Cortés

Published in: Software and Systems Modeling | Issue 1/2017

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Variability testing techniques search for effective and manageable test suites that lead to the rapid detection of faults in systems with high variability. Evaluating the effectiveness of these techniques in realistic settings is a must, but challenging due to the lack of variability-intensive systems with available code, automated tests and fault reports. In this article, we propose using the Drupal framework as a case study to evaluate variability testing techniques. First, we represent the framework variability using a feature model. Then, we report on extensive non-functional data extracted from the Drupal Git repository and the Drupal issue tracking system. Among other results, we identified 3392 faults in single features and 160 faults triggered by the interaction of up to four features in Drupal v7.23. We also found positive correlations relating the number of bugs in Drupal features to their size, cyclomatic complexity, number of changes and fault history. To show the feasibility of our work, we evaluated the effectiveness of non-functional data for test case prioritization in Drupal. Results show that non-functional attributes are effective at accelerating the detection of faults, outperforming related prioritization criteria as test case similarity.
previous article Statistical prioritization for software product line testing: an experience report
next article Analysing the cognitive effectiveness of the WebML visual notation

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now
Appendix
Available only for authorised users
Footnotes
5
A monotonic relationship implies that as the value of one variable increases, so does the value of the other variable; or as the value of one variable increases, the other variable value decreases.
 
Literature
1.
Abal, I., Brabrand, C., Wasowski, A.: 42 variability bugs in the linux kernel: a qualitative analysis. In: International Conference on Automated Software Engineering, pp. 421–432 (2014)
2.
Al-Hajjaji, M., Thum, T., Meinicke, J., Lochau, M., Saake, G.: Similarity-based prioritization in software product-line testing. In: Software Product Line Conference, pp. 197–206 (2014)
3.
Artho, C., Suzaki, K., Di Cosmo, R., Treinen, R., Zacchiroli, S.: Why do software packages conflict? In: Conference on Mining Software Repositories, pp. 141–150, IEEE (2012)
4.
Bagheri, E., Ensan, F., Gasevic, D.: Grammar-based test generation for software product line feature models. In: Conference of the Centre for Advanced Studies on Collaborative Research, pp. 87–101 (2012)
5.
Bagheri, E., Gasevic, D.: Assessing the maintainability of software product line feature models using structural metrics. Softw. Qual. Control. 19(3), 579–612 (2011)
6.
Baller, H., Lity, S., Lochau, M., Schaefer, I.: Multi-objective test suite optimization for incremental product family testing. In: International Conference on Software Testing, Verification, and Validation (2014)
7.
Batory, D.: Feature models, grammars, and propositional formulas. In: Software Product Line Conference (2005)
8.
Benavides, D., Segura, S., Ruiz-Cortés, A.: Automated analyses of feature models 20 years later: a literature review. Inf. Syst. 35(6), 615–636 (2010)CrossRef
9.
Berger, T., She, S., Lotufo, R., Wasowski, A., Czarnecki, K.: Variability modeling in the real: a perspective from the operating systems domain. In: Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, pp. 73–82. ACM (2010)
10.
Berger, T., She, S., Lotufo, R., Wasowski, A., Czarnecki, K.: A study of variability models and languages in the systems software domain. IEEE Trans. Softw. Eng. 39(12), 1611–1640 (2013)CrossRef
11.
12.
13.
14.
Demuth, A., Lopez-Herrejon, R.E., Egyed, A.: Automatic and incremental product optimization for software product lines. In: International Conference on Software Testing, pp. 31–40, IEEE (2014)
15.
Devroey, X., Perrouin, G., Cordy, M., Schobbens, P., Legay, A., Heymans, P.: Towards statistical prioritization for software product lines testing. In: Eighth International Workshop on Variability Modelling of Software-Intensive Systems, Number 10 (2014)
16.
Devroey, X., Perrouin, G., Schobbens, P.: Abstract test case generation for behavioural testing of software product lines. In: Software Product Line Conference, vol. 2, pp. 86–93. ACM (2014)
17.
18.
Elbaum, S., Malishevsky, A.G., Rothermel, G.: Test case prioritization: a family of empirical studies. Trans. Softw. Eng. 28(2), 159–182 (2002)CrossRef
19.
Elbaum, S., Rothermel, G., Kanduri, S., Malishevsky, A.G.: Selecting a cost-effective test case prioritization technique. Softw. Qual. J. 12(3), 185–210 (2004)CrossRef
20.
Ensan, A., Bagheri, E., Asadi, M., Gasevic, D., Biletskiy, Y.: Goal-oriented test case selection and prioritization for product line feature models. In: Conference Information Technology: New Generations, pp. 291–298. IEEE (2011)
21.
Ensan, F., Bagheri, E., Gasevic. D.: Evolutionary search-based test generation for software product line feature models. In: Conference on Advanced Information Systems Engineering, pp. 613–628 (2012)
22.
Ferrer, J., Kruse, P., Chicano, F., Alba, E.: Evolutionary algorithm for prioritized pairwise test data generation. In: Genetic and Evolutionary Computation Conference, pp. 1213–1220 (2012)
23.
Galindo, J.A., Alferez, M., Acher, M., Baudry, B., Benavides, D.: A variability-based testing approach for synthesizing video sequences. In: International Symposium on Software Testing and Analysis, pp. 293–303 (2014)
24.
Galindo, J.A., Benavides, D., Segura, S.: Debian packages repositories as software product line models, towards automated analysis. In: Automated Configuration and Tailoring of Applications, pp. 29–34 (2010)
25.
García-Galán, J., Rana, O., Trinidad, P., Ruiz-Cortés, A.: Migrating to the cloud: a software product line based analysis. In: 3rd International Conference on Cloud Computing and Services Science, pp. 416–426 (2013)
26.
Garvin, B.J., Cohen, M.B.: Feature interaction faults revisited: an exploratory study. In: International Symposium on Software Reliability Engineering, pp. 90–99 (2011)
27.
Garvin, B.J., Cohen, M.B., Dwyer, M.B.: An improved meta-heuristic search for constrained interaction testing. In: International Symposium on Search Based Software Engineering, pp. 13–22 (2009)
28.
Graves, T.L., Karr, A.F., Marron, J.S., Siy, H.: Predicting fault incidence using software change history. Technical report, National Institute of Statistical Sciences, pp. 653–661 (1998)
29.
Guo, J., White, J., Wang, G., Li, J., Wang, Y.: A genetic algorithm for optimized feature selection with resource constraints in software product lines. J. Syst. Softw. 84(12), 2208–2221 (2011)
30.
Hassan, A.E.: Predicting faults using the complexity of code changes. In: International Conference on Software Engineering, pp. 78–88 (2009)
31.
Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., Le Traon, Y.: Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans. Softw. Eng. 40, 1 (2014)CrossRef
32.
Henard, C., Papadakis, M., Perrouin, G., Klein, J., Le Traon, Y.: Multi-objective test generation for software product lines. In: International Software Product Line Conference, pp. 62–71 (2013)
33.
Herzig, K., Just, S., Rau, A., Zeller, A.: Predicting defects using change genealogies. In: International Symposium on Software Reliability Engineering, pp. 118–127 (2013)
34.
Johansen, M.F., Haugen, O., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. Softw. Prod. Line Conf. 1, 46–55 (2012)CrossRef
35.
Johansen, M.F., Haugen, O., Fleurey, F., Eldegard, A.G., Syversen, T.: Generating better partial covering arrays by modeling weights on sub-product lines. In: International Conference on Model Driven Engineering Languages and Systems, pp. 269–284 (2012)
36.
Kang, K., Cohen, S., Hess, J., Novak, W., Peterson, S.: Feature-oriented domain analysis (foda) feasibility study. In: Software Engineering Institute (1990)
37.
Pérez Lamancha, B., Polo Usaola, M.: Testing product generation in software product lines using pairwise for feature coverage. In: International conference on Testing Software and Systems, pp. 111–125 (2010)
38.
Lew, K.S., Dillon, T.S., Forward, K.E.: Software complexity and its impact on software reliability. Trans. Softw. Eng. 14, 1645–1655 (1988)CrossRef
39.
Lopez-Herrejon, R.E., Chicano, F., Ferrer, J., Egyed, A., Alba, E.: Multi-objective optimal test suite computation for software product line pairwise testing. In: IEEE International Conference on Software Maintenance, pp. 404–407 (2013)
40.
Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Egyed, A., Alba, E.: Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of software product lines. In: IEEE Congress on Evolutionary Computation, pp. 387–396 (2014)
41.
Lotufo, R., She, S., Berger, T., Czarnecki, K., Wasowski, A.: Evolution of the linux kernel variability model. In: Software Product Line Conference, pp. 136–150 (2010)
42.
Marijan, D., Gotlieb, A., Sen, S., Hervieu, A.: Practical pairwise testing for software product lines. In: International Software Product Line Conference, pp. 227–235. ACM, New York, NY, USA (2013)
43.
Matsumoto, S., kamei, Y., Monden, A., Matsumoto, K., Nakamura, M.: An analyses of developer metrics for fault prediction. In: International Conference on Predictive Models in Software Engineering, Number 18 (2010)
44.
Mendonca, M., Branco, M., Cowan, D.: S.p.l.o.t.—software product lines online tools. In: Conference Companion on Object Oriented Programming Systems Languages and Applications, pp. 761–762 (2009)
45.
Mohanty, S., Abhinna Acharya, A., Prasad Mohapatra, D.: A survey on model based test case prioritization. Comput. Sci. Inf. Technol. 2, 1042–1047 (2011)
46.
Nagappan, N., Zeller, A., Zimmermann, T., Herzig, K., Murphy, B.: Change bursts as defect predictors. In: International Symposium on Software Reliability Engineering, pp. 309–318 (2010)
47.
Nguyen, H.V., Kästner, C., Nguyen, T.N.: Exploring variability-aware execution for testing plugin-based web applications. In: International Conference on Software Engineering, vol. 6, pp. 907–918 (2014)
48.
Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In: Software Product Line Conference (2010)
49.
Ostrand, T.J., Weyuker, E.J., Bell, R.M.: Where the bugs are. In: International Symposium on Software Testing and Analysis, pp. 86–96 (2004)
50.
Pressmen, S.R.: Software Engineering: A Practitioners Approach. International Edition-5. McGraw Hill, New York (2001)
51.
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2014)
52.
Rothermel, G., Untch, R., Chu, C., Harrold, M.J.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27, 929–948 (2001)CrossRef
53.
Sánchez, A.B., Segura, S.: Automated testing on the analysis of variability-intensive artifacts: an exploratory study with sat solvers. In: XVII Jornadas de Ingeniería del Software y de Bases de Datos (2012)
54.
Sánchez, A.B., Segura, S., Ruiz-Cortés, A.: A comparison of test case prioritization criteria for software product lines. In: IEEE International Conference on Software Testing, Verification, and Validation, pp. 41–50, Cleveland, OH (2014)
55.
Salam Sayyad, A., Menzies, T., Ammar, H.: On the value of user preferences in search-based software engineering: A case study in software product lines. In: International Conference on Software Engineering, pp. 492–501 (2013)
56.
Segura, S., Ruiz-Cortés, A.: Benchmarking on the automated analyses of feature models: A preliminary roadmap. In: International Workshop on Variability Modeling of Software-Intensive Systems, pp. 137–143 (2009)
57.
Segura, S., Sánchez, A.B., Ruiz-Cortés, A.: Automated variability analysis and testing of an e-commerce site: an experience report. In: International Conference on Automated Software Engineering, pp. 139–150. ACM (2014)
58.
She, S., Lotufo, R., Berger, T., Wasowski, A., Czarnecki, K.: The variability model of the linux kernel. In: International Workshop on Variability Modelling of Software-Intensive Systems (2010)
59.
60.
61.
Srikanth, H., Cohen, M.B., Qu, X.: Reducing field failures in system configurable software: cost-based prioritization. In: International Symposium on Software Reliability Engineering, pp. 61–70 (2009)
62.
Svahnberg, M., van Gurp, L., Bosch, J.: A taxonomy of variability realization techniques: research articles. Softw. Pract. Exp. 35, 705–754 (2005)CrossRef
63.
Tan, P.N., Steinbach, M., Kumar, V.: Introduction to Data Mining. Addison Wesley, Boston (2006)
64.
Tiwari, U., Kumar, S.: Cyclomatic complexity metric for component based software. Softw. Eng. Notes 39, 1–6 (2014)
65.
Tomlinson, T., VanDyk, J.K.: Pro Drupal 7 Development, 3rd edn (2010)
66.
Wang, S., Ali, S., Gotlieb, A.: Minimizing test suites in software product lines using weight-based genetic algorithms. In: The Genetic and Evolutionary Computation Conference, pp. 1493–1500 (2013)
67.
Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D., Liaaen, M.: Multi-objective test prioritization in software product line testing: an industrial case study. In: Software Product Line Conference, pp. 32–41 (2014)
68.
Yoo, S., Harman, M.: Regression testing minimisation, selection and prioritisation: a survey. In: Software Testing, Verification and Reliability, pp. 67–120 (2012)
Metadata
Title
Variability testing in the wild: the Drupal case study
Authors
Ana B. Sánchez
Sergio Segura
José A. Parejo
Antonio Ruiz-Cortés
Publication date
19-04-2015
Publisher
Springer Berlin Heidelberg
Published in
Software and Systems Modeling / Issue 1/2017
Print ISSN: 1619-1366
Electronic ISSN: 1619-1374
DOI
https://doi.org/10.1007/s10270-015-0459-z

Other articles of this Issue 1/2017

Software and Systems Modeling 1/2017 Go to the issue

Special Section Paper

Constraint-based test generation for automotive operating systems

Regular Paper

An empirical approach toward the resolution of conflicts in goal-oriented models

Regular Paper

Eugenia: towards disciplined and automated development of GMF-based graphical model editors

Special Section Paper

Inferring physical units in formal models

Theme Section Paper

Analysing the Linux kernel feature model changes using FMDiff

Erratum

Erratum to: Introduction to the theme issue on variability modeling of software-intensive systems

Premium Partner

    Image Credits