Top

Published in:

03-02-2018

Do software models based on the UML aid in source-code comprehensibility? Aggregating evidence from 12 controlled experiments

Authors: Giuseppe Scanniello, Carmine Gravino, Marcela Genero, José A. Cruz-Lemus, Genoveffa Tortora, Michele Risi, Gabriella Dodero

Published in: Empirical Software Engineering | Issue 5/2018

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In this paper, we present the results of long-term research conducted in order to study the contribution made by software models based on the Unified Modeling Language (UML) to the comprehensibility of Java source-code deprived of comments. We have conducted 12 controlled experiments in different experimental contexts and on different sites with participants with different levels of expertise (i.e., Bachelor’s, Master’s, and PhD students and software practitioners from Italy and Spain). A total of 333 observations were obtained from these experiments. The UML models in our experiments were those produced in the analysis and design phases. The models produced in the analysis phase were created with the objective of abstracting the environment in which the software will work (i.e., the problem domain), while those produced in the design phase were created with the goal of abstracting implementation aspects of the software (i.e., the solution/application domain). Source-code comprehensibility was assessed with regard to correctness of understanding, time taken to accomplish the comprehension tasks, and efficiency as regards accomplishing those tasks. In order to study the global effect of UML models on source-code comprehensibility, we aggregated results from the individual experiments using a meta-analysis. We made every effort to account for the heterogeneity of our experiments when aggregating the results obtained from them. The overall results suggest that the use of UML models affects the comprehensibility of source-code, when it is deprived of comments. Indeed, models produced in the analysis phase might reduce source-code comprehensibility, while increasing the time taken to complete comprehension tasks. That is, browsing source code and this kind of models together negatively impacts on the time taken to complete comprehension tasks without having a positive effect on the comprehensibility of source code. One plausible justification for this is that the UML models produced in the analysis phase focus on the problem domain. That is, models produced in the analysis phase say nothing about source code and there should be no expectation that they would, in any way, be beneficial to comprehensibility. On the other hand, UML models produced in the design phase improve source-code comprehensibility. One possible justification for this result is that models produced in the design phase are more focused on implementation details. Therefore, although the participants had more material to read and browse, this additional effort was paid back in the form of an improved comprehension of source code.

previous article Studying software logging using topic models

next article Program comprehension of domain-specific and general-purpose languages: replication of a family of experiments using integrated development environments

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

An instance of a process is a sequence of the activities/phases.

It is also called requirements engineering process and it is the process of determining user expectations (i.e., requirements) as regards a new or modified product.

It maps the requirements onto the software architecture that defines the components, their interfaces and behaviors. The design document describes a plan with which to implement the requirements.

Design-pattern instances can be seen as a micro-architecture that developers copy and adapt to their particular designs in order to solve the recurrent problem described by the design pattern (Bruegge and Dutoit 2003; Gamma et al. 1995).

www2.unibas.it/gscanniello/SourceCodeComprMetaAnalysis/data.xlsx

This is a software system that is used to sell and manage CDs/DVDs in a music shop. The feature search for a singer was used in the experiments: the user inserts a string (e.g., the surname of the singer), and the system then searches for all the singers that satisfy the search criterion and shows them in a list of the associated information.

This is a software system with which to book and buy theater tickets. The feature buy a theater ticket was used in the experiments: the system shows the list of the available tickets for a given theater and performance, and the user then chooses the ticket and inserts data about the spectator.

The time was an approximation of comprehension effort. This complies with the ISO/IEC 9126 standard ISO (1991) (and subsequent versions), in which effort is the productive time associated with a specific project task.

If a participant is tested first under the experimental condition A and then under the experimental condition B, she/he could potentially perform better or worse under condition B.

Efficiency is a derived measure that is computed as the ratio between comprehension and completion time. Task efficiency is a ratio measure and estimates a participant’s efficiency as regards the execution of a comprehension task. The larger the efficiency value, the better it is. The perspective we adopted is that of quality in use (e.g., ISO 2000, 2011), since efficiency measures source-code comprehension achieved during the expenditure of available models.

Effect size is statistically different from the overall effect if the diamond (at the bottom of the forest plot) does not intersect the vertical line.

This is a range of values that we are 95% certain that it contains the true mean value.

Abrahão SM, Gravino C, Pelozo EI, Scanniello G, Tortora G (2013) Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: Results from a family of five experiments. IEEE Trans Softw Eng 39 (3):327–342CrossRef

Agarwal R, Sinha AP (2003) Object-oriented modeling with UML: a study of developers’ perceptions. Commun ACM 46(9):248–256CrossRef

Anda B, Hansen K, Gullesen I, Thorsen HK (2006) Experiences from introducing UML-based development in a large safety-critical project. Empir Softw Eng 11(4):555–581CrossRef

Antoniol G, Canfora G, Casazza G, De Lucia A, Merlo E (2002) Recovering traceability links between code and documentation. IEEE Trans Softw Eng 28(10):970–983CrossRef

Arisholm E, Briand LC, Hove SE, Labiche Y (2006) The impact of UML documentation on software maintenance: An experimental evaluation. IEEE Trans Softw Eng 32(6):365–381CrossRef

Basili VR, Rombach HD (1988) The TAME project: Towards improvement-oriented software environments. IEEE Trans Softw Eng 14(6):758–773CrossRef

Basili V, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473CrossRef

Bavota G, Canfora G, Di Penta M, Oliveto R, Panichella S (2013) An empirical investigation on documentation usage patterns in maintenance tasks. In: Proceedings of International Conference on Software Maintenance. IEEE Computer Society, pp 210–219

Bruegge B, Dutoit AH (2003) Object-oriented software engineering: using UML, Patterns and Java, 2nd edn. Prentice-Hall, Upper Saddle River

Budgen D, Burn AJ, Brereton OP, Kitchenham B, Pretorius R (2011) Empirical evidence about the UML: a systematic literature review. Softw Pract Exper 41(4):363–392CrossRef

Cariou E, Beugnard A, Jezequel JM (2002) An architecture and a process for implementing distributed collaborations. In: Proceedings of International Enterprise Distributed Object Computing, pp 132–143

Carver J, Jaccheri L, Morasca S, Shull F (2003) Issues in using students in empirical studies in software engineering education. In: Proceedings of International Symposium on Software Metrics. IEEE Computer Society, pp 239–250

Corazza A, Maggio V, Scanniello G (2016) Coherence of comments and method implementations: a dataset and an empirical investigation. Softw Q J:1–27. https://doi.org/10.1007/s11219-016-9347-1

Dzidek WJ, Arisholm E, Briand LC (2008) A realistic empirical evaluation of the costs and benefits of UML in software maintenance. IEEE Trans Softw Eng 34 (3):407–432CrossRef

Eclipse Modeling Framework (EMF) (2012) http://www.eclipse.org/modeling/emf/

Erickson J, Siau K (2007) Theoretical and practical complexity of modeling methods. Commun ACM 50(8):46–51CrossRef

Fernȧndez-Sȧez AM, Chaudron MRV, Genero M (2013) Exploring costs and benefits of using UML on maintenance: Preliminary findings of a case study in a large IT department. In: Proceedings of the International Workshop on Experiences and Empirical Studies in Software Modeling co-located with the International Conference on Model Driven Engineering Languages and Systems, pp 33–42

Fernȧndez-Sȧez AM, Caivano D, Genero M, Chaudron MRV (2015) On the use of UML documentation in software maintenance: Results from a survey in industry. In: Proceedings of ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp 292–301

Fernȧndez-Sȧez AM, Genero M, Caivano D, Chaudron MRV (2016) Does the level of detail of UML diagrams affect the maintainability of source code?: a family of experiments. Empir Softw Eng 21(1):212–259CrossRef

Fernȧndez-Sȧez AM, Genero M, Chaudron MRV (2013) Empirical studies concerning the maintenance of UML diagrams and their use in the maintenance of code: A systematic mapping study. Inf Softw Technol 55(7):1119–1142CrossRef

Fernȧndez-Sȧez AM, Genero M, Chaudron MRV, Caivano D, Ramos I (2015) Are forward designed or reverse-engineered UML diagrams more helpful for code maintenance?: A family of experiments. Inf Softw Technol 57:644–663CrossRef

Fluri B, Wursch M, Gall H (2007) Do code and comments co-evolve? on the relation between source code and comment changes. In: Proceedings of the Working Conference on Reverse Engineering. IEEE Computer Society, pp 70–79

Fu R, Gartlehner G, Grant M, Shamliyan T, Sedrakyan A, Wilt TJ, Griffith L, Oremus M, Raina P, Ismaila A, Santaguida P, Lau J, Trikalinos TA (2011) Conducting quantitative synthesis when comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol 64(11):1187–1197CrossRef

Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object oriented software. Addison-Wesley, BostonMATH

Garousi G, Garousi V, Moussavi M, Ruhe G, Smith B (2013) Evaluating usage and quality of technical software documentation: an empirical study. In: Proceedings of International Conference on Evaluation and Assessment in Software Engineering, pp 24–35

Gravino C, Tortora G, Scanniello G (2010) An empirical investigation on the relation between analysis models and source code comprehension. In: Proceedings of the International Symposium on Applied Computing. ACM, pp 2365–2366

Gravino C, Scanniello G, Tortora G (2015) Source-code comprehension tasks supported by UML design models: Results from a controlled experiment and a differentiated replication. J Vis Lang Comput 28:23–38CrossRef

Grossman M, Aronson JE, McCarthy RV (2005) Does UML make the grade? Insights from the software development community. Inf Softw Technol 47(6):383–397CrossRef

Guéhéneuc YG (2007) P-mart: Pattern-like micro architecture repository. In: Proceedings of EuroPLoP Focus Group on Pattern Repositories

Hammad M, Collard ML, Maletic JI (2011) Automatically identifying changes that impact code-to-design traceability during evolution. Softw Qual J 19(1):35–64CrossRef

Higgins JPT, Green S (2008) Cochrane handbook for systematic reviews of interventions, 5th edn. The Cochrane Collaboration, LondonCrossRef

Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella J (2006) Assessing heterogeneity in meta-analysis: Q statistic or i2 index? Psychol Methods 11 (2):193–206CrossRef

Hutchinson JE, Whittle J, Rouncefield M, Kristoffersen S (2011) Empirical assessment of MDE in industry. In: Proceedings of the International Conference on Software Engineering, pp 471–480

ISO (1991) Information Technology–Software Product Evaluation: Quality Characteristics and Guidelines for their Use ISO/IEC IS 9126. ISO, Geneva

ISO (2000) ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) – Part 9: Requirements for non-keyboard input devices. ISO, Geneva

ISO (2011) ISO/IEC 25010 Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. ISO, Geneva

Jedlitschka A, Ciolkowski M, Pfahl D, Sjoberg D (2008) Reporting experiments in software engineering. In: Shull F, Singer J (eds) Guide to Advanced Empirical Software Engineering, Springer, pp 201–228

Jiang ZM, Hassan AE (2006) Examining the evolution of code comments in postgresql. In: Proceedings of Mining Software Repositories. ACM, pp 179–180

Juristo N, Moreno A (2001) Basics of software engineering experimentation. Kluwer Academic Publishers, DordrechtCrossRefMATH

Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734CrossRef

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering

Lehnert S, Farooq Q u a, Riebisch M (2013) Rule-based impact analysis for heterogeneous software artifacts. In: Proceedings of the European Conference on Software Maintenance and Reengineering, pp 209–218

Leotta M, Ricca F, Antoniol G, Garousi V, Zhi J, Ruhe G (2013) A pilot experiment to quantify the effect of documentation accuracy on maintenance tasks. In: Proceedings of International Conference on Software Maintenance, pp 428–431

OMG (2005) Unified modeling language (UML) specification version 2.0. Technical report, Object Management Group

Oppenheim AN (1992) Questionnaire design, interviewing and attitude measurement, Pinter, London

Pavalkis S, Nemuraite L (2013) Process for applying derived property based traceability framework in software and systems development life cycle. Springer, Berlin Heidelberg, pp 122–133

Pavalkis S, Nemuraite L, Butkiene R (2013) Derived properties: A user friendly approach to improving model traceability. Inf Technol Control 42(1):48–60

Pickard L, Kitchenham B, Jones P (1998) Combining empirical results in software engineering. Inf Softw Technol 40(14):811–821CrossRef

Ried K (2008) Interpreting and understanding meta-analysis graphs - A practical guide, vol 35. Australian College of General Practitioners, East Melbourne

Salviulo F, Scanniello G (2014) Dealing with identifiers and comments in source code comprehension and maintenance: Results from an ethnographically-informed study with students and professionals. In: Proceedings of International Conference on Evaluation and Assessment in Software Engineering. ACM

Scanniello G, Gravino C, Risi M, Tortora G (2010) A controlled experiment for assessing the contribution of design pattern documentation on software maintenance. In: Proceedings of the Symposium on Empirical Software Engineering and Measurement. ACM

Scanniello G, Gravino C, Tortora G (2010) Investigating the role of UML in the software modeling and maintenance - a preliminary industrial survey. In: Proceedings of International Conference on Enterprise Information Systems, pp 141–148

Scanniello G, Gravino C, Genero M, Cruz-Lemus JA, Tortora G (2014) On the impact of UML analysis models on source code comprehensibility and modifiability. ACM Trans Softw Eng Methodol 23(2):13:1–13:26CrossRef

Scanniello G, Gravino C, Risi M, Tortora G, Dodero G (2015) Documenting design-pattern instances: A family of experiments on source-code comprehensibility. ACM Trans Softw Eng Methodol 24(3):14CrossRef

Scanniello G, Gravino C, Tortora G, Genero M, Risi M, Cruz-Lemus JA, Dodero G (2015) Studying the effect of uml-based models on source-code comprehensibility: Results from a long-term investigation. In: Springer (ed.) Proceedings of International Conference on Product-Focused Software Process Improvement, vol 9459. Lecture Notes in Computer Science, pp 311–327

Settimi R, Cleland-Huang J, Khadra OB, Mody J, Lukasik W, DePalma C (2004) Supporting software evolution through dynamically retrieving traces to uml artifacts. In: Proceedings of International Workshop on Principles of Software Evolution, pp 49–54

Shull F, Carver JC, Vegas S, Juzgado NJ (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218CrossRef

Sillito J, Murphy GC, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451CrossRef

Tang A, Jin Y, Han J (2007) A rationale-based architecture model for design traceability and reasoning. J Syst Softw 80(6):918–934CrossRef

Tang A, Nicholson A, Jin Y, Han J (2007) Using bayesian belief networks for change impact analysis in architecture design. J Syst Softw 80(1):127–148CrossRef

Wohlin C, Runeson P, Höst M., Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, BerlinCrossRefMATH

Zhi J, Sun B, Garousi G, Shahnewaz SM, Ruhe G (2015) Cost, benefits and quality of software development documentation: A systematic mapping. J Syst Softw 99:175–198CrossRef

Title: Do software models based on the UML aid in source-code comprehensibility? Aggregating evidence from 12 controlled experiments
Authors: Giuseppe Scanniello
Carmine Gravino
Marcela Genero
José A. Cruz-Lemus
Genoveffa Tortora
Michele Risi
Gabriella Dodero
Publication date: 03-02-2018
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 5/2018
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-017-9591-4

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 5/2018

Revisiting the performance of automated approaches for the retrieval of duplicate reports in issue tracking systems that perform just-in-time duplicate retrieval

Studying software logging using topic models

Overfitting in semantics-based automated program repair

Pinpointing and repairing performance bottlenecks in concurrent programs

Getting the most from map data structures in Android

Augmenting and structuring user queries to support efficient free-form code search

Premium Partner