Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Bücher
Zeitschriften
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
MTZ-Fachkongress

Antriebe und Energiesysteme von morgen


Austausch von Automobil- und Energiewirtschaft

Am 14./15. Mai geht es in Chemnitz um die Mobilitätswende durch Elektro- und Wasserstofffahrzeuge.
Erweiterte Suche
Anmelden
nach oben
Empirical Software Engineering
Erschienen in: Empirical Software Engineering 1/2024

01.02.2024

What makes a code review useful to OpenDev developers? An empirical investigation

verfasst von: Asif Kamal Turzo, Amiangshu Bosu

Erschienen in: Empirical Software Engineering | Ausgabe 1/2024

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Context

Due to the association of significant efforts, even a minor improvement in the effectiveness of Code Reviews(CR) can incur significant savings for a software development organization.

Objective

This study aims to develop a finer grain understanding of what makes a code review comment useful to OSS developers, to what extent a code review comment is considered useful to them, and how various contextual and participant-related factors influence its degree of usefulness.

Method

On this goal, we have conducted a three-stage mixed-method study. We randomly selected 2,500 CR comments from the OpenDev Nova project and manually categorized the comments. We designed a survey of OpenDev developers to better understand their perspectives on useful CRs. Combining our survey-obtained scores with our manually labeled dataset, we trained two regression models - one to identify factors that influence the usefulness of CR comments and the other to identify factors that improve the odds of ‘Functional’ defect identification over the others.

Results

The results of our study suggest that a CR comment’s usefulness is dictated not only by its technical contributions, such as defect findings or quality improvement tips but also by its linguistic characteristics, such as comprehensibility and politeness. While a reviewer’s coding experience is positively associated with CR usefulness, the number of mutual reviews, comment volume in a file, the total number of lines added /modified, and CR interval have the opposite associations. While authorship and reviewership experiences for the files under review have been the most popular attributes for reviewer recommendation systems, we do not find any significant association of those attributes with CR usefulness.

Conclusion

We recommend discouraging frequent code review associations between two individuals as such associations may decrease CR usefulness. We also recommend authoring CR comments in a constructive and empathetic tone. As several of our results deviate from prior studies, we recommend more investigations to identify context-specific attributes to build reviewer recommendation models.
Vorheriger Artikel Detecting outdated code element references in software repository documentation
Nächster Artikel Does code review speed matter for practitioners?

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren
Fußnoten
3
Kappa (\(\kappa \)) scores are commonly interpreted as: 0.01-0.20 as ‘none to slight,’ 0.21-0.40 as ‘fair,’ 0.41- 0.60 as ‘moderate,’ 0.61-0.80 as ‘substantial,’ and 0.81-1.00 as ‘almost perfect agreement’ Landis and Koch (1977).
 
4
\(p-values\) are adjusted using Benjamini and Hochberg corrections (Benjamini and Hochberg 1995) due to multiple comparisons.
 
Literatur
Allison P (2014) Prediction vs. causation in regression analysis. Statistical Horizons 703
Asthana S, Kumar R, Bhagwan R, Bird C, Bansal C, Maddila C, Mehta S, Ashok B (2019) Whodo: automating reviewer suggestions at scale. In: Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 937–945
Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: 2013 35th international conference on software engineering (ICSE), IEEE pp 712–721
Balachandran V (2013) Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In: 2013 35th International conference on software engineering (ICSE), IEEE pp 931–940
Baltes S, Diehl S (2016) Worse than spam: Issues in sampling software developers. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement, pp 1–6
Barnett M, Bird C, Brunet J, Lahiri SK (2015) Helping developers help themselves: Automatic decomposition of code review changesets. In: Proceedings of the 37th international conference on software engineering- vol 1, IEEE Press, pp 134–144
Baum T, Schneider K (2016) On the need for a new generation of code review tools. In: International conference on product-focused software process improvement, Springer, pp 301–308
Bayaga A (2010) Multinomial logistic regression: Usage and application in risk analysis. J App Quantitative Methods 5(2)
Beller M, Bacchelli A, Zaidman A, Juergens E (2014) Modern code reviews in open-source projects: Which problems do they fix? In: Proceedings of the 11th working conference on mining software repositories, pp 202–211
Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J Roy Stat Soc: Ser B (Methodol) 57(1):289–300MathSciNetCrossRef
Bosu A, Carver JC, Bird C, Orbeck J, Chockley C (2017) Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft. IEEE Trans Software Eng 43(1):56–75CrossRef
Bosu A, Carver JC, Hafiz M, Hilley P, Janni D (2014) Identifying the characteristics of vulnerable code changes: An empirical study. 22nd ACM SIGSOFT international symposium on the foundations of software engineering, FSE ’14. China, Hong Kong, pp 257–268
Bosu A, Greiler M, Bird C (2015) Characteristics of useful code reviews: An empirical study at microsoft. In: 2015 IEEE/ACM 12th Working conference on mining software repositories, IEEE pp 146–156
Changyong F, Hongyue W, Naiji L, Tian C, Hua H, Ying L et al (2014) Log-transformation and its implications for data analysis. Shanghai Arch Psychiatry 26(2):105
Chouchen M, Ouni A, Mkaouer MW, Kula RG, Inoue K (2021) Whoreview: A multi-objective search-based approach for code reviewers recommendation in modern code review. Appl Soft Comput 100:106908CrossRef
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Measur 20(1):37–46CrossRef
Czerwonka J, Greiler M, Tilford J (2015) Code reviews do not find bugs. how the current code review best practice slows us down. In: 2015 IEEE/ACM 37th IEEE International conference on software engineering, vol 2, IEEE, pp 27–28
Dias M, Bacchelli A, Gousios G, Cassou D, Ducasse S (2015) Untangling fine-grained code changes. In: 2015 IEEE 22nd International conference on software analysis, evolution, and reengineering (SANER), IEEE, pp 341–350
Ebert F, Castor F, Novielli N, Serebrenik A (2021) An exploratory study on confusion in code reviews. Empir Softw Eng 26:1–48CrossRef
Fowler M, Highsmith J et al (2001) The agile manifesto. Software Development 9(8):28–35
Fukushima T, Kamei Y, McIntosh S, Yamashita K, Ubayashi N (2014) An empirical study of just-in-time defect prediction using cross-project models. In: Proceedings of the 11th working conference on mining software repositories, pp 172–181
Gauthier IX, Lamothe M, Mussbacher G, McIntosh S (2021) Is historical data an appropriate benchmark for reviewer recommendation systems?: A case study of the gerrit community. In: 2021 36th IEEE/ACM International conference on automated software engineering (ASE), IEEE, pp 30–41
Gómez VU, Ducasse S, D’Hondt T (2015) Visually characterizing source code changes. Sci Comput Program 98:376–393CrossRef
Gunawardena SD, Devine P, Beaumont I, Garden LP, Murphy-Hill E, Blincoe K (2022) Destructive criticism in software code review impacts inclusion. Proceedings of the ACM on Human-Computer Interaction 6(CSCW2):1–29CrossRef
Han X, Tahir A, Liang P, Counsell S, Luo Y (2021) Understanding code smell detection via code review: A study of the openstack community. In: 2021 IEEE/ACM 29th International conference on program comprehension (ICPC), IEEE, pp 323–334
Harrell FE Jr, Lee KL, Califf RM, Pryor DB, Rosati RA (1984) Regression modelling strategies for improved prognostic prediction. Stat Med 3(2):143–152CrossRef
Hasan M, Iqbal A, Islam MRU, Rahman A, Bosu A (2021) Using a balanced scorecard to identify opportunities to improve code review effectiveness: an industrial experience report. Empir Softw Eng 26(6):1–34CrossRef
Hassan AE, Holt RC (2003) Studying the chaos of code development. In: WCRE, vol 3, p 123
Hatton L (2008) Testing the value of checklists in code inspections. IEEE Softw 25(4):82–88CrossRef
Helland IS (1987) On the interpretation and use of r2 in regression analysis. Biometrics pp 61–69
Henley AZ, Muçlu K, Christakis M, Fleming SD, Bird C (2018) Cfar: A tool to increase communication, productivity, and review quality in collaborative code reviews. In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp 1–13
Hinkle D, Jurs H, Wiersma W (1998) Applied statistics for the behavioral sciences
Hirao T, Ihara A, Ueda Y, Phannachitta P, Matsumoto Ki (2016) The impact of a low level of agreement among reviewers in a code review process. In: IFIP International conference on open source systems, Springer, pp 97–110
Hirao T, McIntosh S, Ihara A, Matsumoto K (2020) Code reviews with divergent review scores: An empirical study of the openstack and qt communities. IEEE Trans Softw Eng
Hong Y, Tantithamthavorn C, Thongtanunam P, Aleti A (2022) Commentfinder: a simpler, faster, more accurate code review comments recommendation. In: Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, pp 507–519
Jiarpakdee J, Tantithamthavorn C, Treude C (2018) Autospearman: Automatically mitigating correlated software metrics for interpreting defect models. In: 2018 IEEE International conference on software maintenance and evolution (ICSME), IEEE computer society pp 92–103
Kincaid JP, Fishburne Jr RP, Rogers RL, Chissom BS (1975) Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Tech rep, naval technical training command millington tn research branch
Kononenko O, Baysal O, Guerrouj L, Cao Y, Godfrey MW (2015) Investigating code review quality: Do people and participation matter? In: 2015 IEEE international conference on software maintenance and evolution (ICSME), IEEE, pp 111–120
Kononenko O, Rose T, Baysal O, Godfrey M, Theisen D, De Water B (2018) Studying pull request merges: a case study of shopify’s active merchant. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, pp 124–133
Landis JR, Koch GG (1977) An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics pp 363–374
Mansfield ER, Helms BP (1982) Detecting multicollinearity. Am Stat 36(3a):158–160CrossRef
Mäntylä MV, Lassenius C (2008) What types of defects are really discovered in code reviews? IEEE Trans Softw Eng 35(3):430–448CrossRef
McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21:2146–2189CrossRef
Mirsaeedi E, Rigby PC (2020) Mitigating turnover with code review recommendation: balancing expertise, workload, and knowledge distribution. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, pp 1183–1195
Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Proceedings of the 27th international conference on Software engineering, pp 284–292
Nagappan N, Ball T (2007) Using software dependencies and churn metrics to predict field failures: An empirical case study. In: First international symposium on empirical software engineering and measurement (ESEM 2007), IEEE, pp 364–373
Nagelkerke NJ et al (1991) A note on a general definition of the coefficient of determination. Biometrika 78(3):691–692MathSciNetCrossRef
Osborne JW, Waters E (2002) Four assumptions of multiple regression that researchers should always test. Pract Assess Res Eval 8(1):2
Pandya P, Tiwari S (2022) Corms: A github and gerrit based hybrid code reviewer recommendation approach for modern code review. In: Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, ESEC/FSE 2022, association for computing machinery, New York, NY, USA p 546-557
Panichella S, Zaugg N (2020) An empirical investigation of relevant changes and automation needs in modern code review. Empirical Software Engineering 25(TBD), TBD
Paul R, Turzo AK, Bosu A (2021) Why security defects go unnoticed during code reviews? a case-control study of the chromium os project. In: 2021 IEEE/ACM 43rd International conference on software engineering (ICSE), IEEE pp 1373–1385
Pinto G, Steinmacher I, Gerosa MA (2016) More common than you think: An in-depth study of casual contributors. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), vol 1, IEEE, pp 112–123
Rahman MM, Roy CK, Collins JA (2016) Correct: code reviewer recommendation in github based on cross-project and technology experience. In: Proceedings of the 38th International conference on software engineering companion, pp 222–231
Rahman MM, Roy CK, Kula RG (2017) Predicting usefulness of code review comments using textual features and developer experience. In: 2017 IEEE/ACM 14th International conference on mining software repositories (MSR), IEEE, pp 215–226
Rigby PC, Bird C (2013) Convergent contemporary software peer review practices. In: Proceedings of the 2013 9th joint meeting on foundations of software engineering, pp 202–212
Rigby PC, Storey MA (2011) Understanding broadcast based peer review on open source software projects. In: 2011 33rd International conference on software engineering (ICSE), IEEE, pp 541–550
Rong G, Zhang Y, Yang L, Zhang F, Kuang H, Zhang H (2022) Modeling review history for reviewer recommendation: A hypergraph approach. In: Proceedings of the 44th International conference on software engineering, ICSE ’22, association for computing machinery, New York, NY, USA p 1381–1392
Sadowski C, Söderberg E, Church L, Sipko M, Bacchelli A (2018) Modern code review: a case study at google. In: Proceedings of the 40th International conference on software engineering: software engineering in practice, pp 181–190
Sarker J, Turzo AK, Bosu A (2020) A benchmark study of the contemporary toxicity detectors on software engineering interactions. arXiv preprint arXiv:​2009.​09331
Sarle W (1990) Sas/stat user’s guide: The varclus procedure. sas institute. Inc. Cary, NC, USA
Snow J, Mann M (2013) Qualtrics survey software: handbook for research professionals. Qualtrics Labs, Inc
Tao Y, Kim S (2015) Partitioning composite code changes to facilitate code review. In: 2015 IEEE/ACM 12th working conference on mining software repositories, IEEE, pp 180–190
Thongtanunam P, Hassan AE (2020) Review dynamics and their impact on software quality. IEEE Trans Software Eng 47(12):2698–2712CrossRef
Thongtanunam P, McIntosh S, Hassan AE, Iida H (2015) Investigating code review practices in defective files: An empirical study of the qt system. In: 2015 IEEE/ACM 12th Working conference on mining software repositories, IEEE, pp 168–179
Thongtanunam P, McIntosh S, Hassan AE, Iida H (2017) Review participation in modern code review. Empir Softw Eng 22(2):768–817CrossRef
Thongtanunam P, Pornprasit C, Tantithamthavorn C (2022) Autotransform: automated code transformation to support modern code review process. In: Proceedings of the 44th international conference on software engineering, pp 237–248
Thongtanunam P, Tantithamthavorn C, Kula RG, Yoshida N, Iida H, Matsumoto Ki (2015) Who should review my code? a file location-based code-reviewer recommendation approach for modern code review. In: 2015 IEEE 22nd International conference on software analysis, evolution, and reengineering (SANER), IEEE, pp 141–150
Tufan R, Pascarella L, Tufanoy M, Poshyvanykz D, Bavota G (2021) Towards automating code review activities. In: 2021 IEEE/ACM 43rd International conference on software engineering (ICSE), IEEE, pp 163–174
Tufano R, Masiero S, Mastropaolo A, Pascarella L, Poshyvanyk D, Bavota G (2022) Using pre-trained models to boost code review automation. In: Proceedings of the 44th international conference on software engineering, pp 2291–2302
Yamane T (1973) Statistics: an introductory analysis-3
Zaidman A, Van Rompaey B, Van Deursen A, Demeyer S (2011) Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empir Softw Eng 16:325–364CrossRef
Zanaty FE, Hirao T, McIntosh S, Ihara A, Matsumoto K (2018) An empirical study of design discussions in code review. In: Proceedings of the 12th ACM/IEEE International symposium on empirical software engineering and measurement, pp 1–10
Zanjani MB, Kagdi H, Bird C (2015) Automatically recommending peer reviewers in modern code review. IEEE Trans Software Eng 42(6):530–543
Metadaten
Titel
What makes a code review useful to OpenDev developers? An empirical investigation
verfasst von
Asif Kamal Turzo
Amiangshu Bosu
Publikationsdatum
01.02.2024
Verlag
Springer US
Erschienen in
Empirical Software Engineering / Ausgabe 1/2024
Print ISSN: 1382-3256
Elektronische ISSN: 1573-7616
DOI
https://doi.org/10.1007/s10664-023-10411-x

Weitere Artikel der Ausgabe 1/2024

Empirical Software Engineering 1/2024 Zur Ausgabe

OriginalPaper

Predicting merge conflicts considering social and technical assets

OriginalPaper

Which design decisions in AI-enabled mobile applications contribute to greener AI?

OriginalPaper

The Impact of Personality on Requirements Engineering Activities: A Mixed-Methods Study

OriginalPaper

Mutation analysis for evaluating code translation

OriginalPaper

A fly in the ointment: an empirical study on the characteristics of Ethereum smart contract code weaknesses

OriginalPaper

SAFe transformation in a large financial corporation

Premium Partner

    Bildnachweise