Top

Empirical Software Engineering

Published in:

01-09-2023

Do attention and memory explain the performance of software developers?

Authors: Valentina Piantadosi, Simone Scalabrino, Alexander Serebrenik, Nicole Novielli, Rocco Oliveto

Published in: Empirical Software Engineering | Issue 5/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Writing and modifying source code are core activities in software development and evolution. The outcome of a coding task in terms of quality may depend on several aspects, such as the difficulty of the task or the complexity of the system. Besides, it is well known that individual characteristics of developers, like the programming experience, play a lead role in this. Recent work started exploring the influence that cognitive human aspects have on the ability of developers to acquire information from the source code (e.g., finding security blind spots). However, it is still unknown to what extent such aspects influence their ability of completing coding tasks. In this paper, we theorize that two cognitive human aspects, attention and memory, play a role in predicting the outcome of a coding task. We conducted a controlled experiment involving 32 participants (18 bachelor students, 9 master students, 2 Ph.D. students. and 3 practitioners), in which we asked them to complete two bug-fixing and two feature implementation tasks. We measured, for each of them, three attention-related factors (i.e., alerting, orienting, and executive control) and two memory-related ones (i.e., working memory and immediate recall) through well-established psychometric tests. Finally, we investigated to what extent these factors can explain the correctness, the readability and the time taken to complete a task in function of such factors. Our results show that all the attention- and memory-related factors achieved very low correlation with correctness and time. Indeed, the number of years of programming experience is far more important than all the other variables we considered for explaining the correctness and the time required to complete a task. Moreover, we found a significant relationship between orienting (an attention-related factor) and code readability.

previous article A comparison of reinforcement learning frameworks for software testing tasks

next article On practitioners’ concerns when adopting service mesh frameworks

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

ANT is also known as “Flanker Inhibitory Control and Attention Test” (e.g., in the NIH Toolbox Cognition Battery))

https://leetcode.com/

We used the tool available at https://www.statskingdom.com/sample_size_regression.html

See, for example: https://github.com/FantacyWorld/libMSVL/blob/master/dyn_array.c#L307.

Allan J (2013) Cognitions, Springer New York, pp 441–441. https://doi.org/10.1007/978-1-4419-1005-9_1114

Baddeley AD (1983) Working memory. Philos Trans R Soc Lond of London B, Biol Sci 302(1110):311–324CrossRef

Benedict RH, DeLuca J, Phillips G, LaRocca N, Hudson LD, Rudick R, Consortium MSOA (2017) Validity of the symbol digit modalities test as a cognition performance outcome measure for multiple sclerosis. Mult Scler J 23(5):721–733CrossRef

Benjafield JG, Smilek D, Kingstone A (2010) Cognition, 4th edn. Oxford University Press, New York

Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B (Methodol) 57(1):289–300MathSciNetMATH

Bhattacharya P, Neamtiu I (2011) Bug-fix time prediction models: can we do better? In: Proceedings of the 8th working conference on mining software repositories, pp 207–210

Bialystok E, DePape AM (2009) Musical expertise, bilingualism, and executive functioning. J Exp Psychol Hum Percept Perform 35(2):565CrossRef

Blackburn JD, Scudder GD, Van Wassenhove LN (1996) Improving speed and productivity of software development: a global survey of software developers. IEEE Trans Soft Eng 22(12):875–885CrossRef

Blank A, Frush Holt R, Pisoni DB, Kronenberger WG (2020) Associations between parenting stress, language comprehension, and inhibitory control in children with hearing loss. J Speech Hear Res 63(1):321–333CrossRef

Bowrin AR, King J (2010) Time pressure, task complexity, and audit effectiveness. Managerial auditing journal

Brearly TW, Rowland JA, Martindale SL, Shura RD, Curry D, Taber KH (2018) Comparability of ipad and web-based nih toolbox cognitive battery administration in veterans. Arch Clin Neuropsychol 34(4):524–530CrossRef

Brun Y, Lin T, Somerville JE, Myers E, Ebner NC (2021) Blindspots in python and java apis result in vulnerable code. arXiv:2103.06091

Cairncross M, Gindwani H, Rita Egbert A, Torres IJ, Hutchison JS, Dams O’Connor K, Panenka WJ, Brubacher JR, Meddings L, Kwan L, et al (2022) Criterion validity of the brief test of adult cognition by telephone (btact) for mild traumatic brain injury. Brain Injury pp 1–9

Cappos J, Zhuang Y, Oliveira D, Rosenthal M, Yeh KC (2014) Vulnerabilities as blind spots in developer’s heuristic-based decision-making processes. In: Proceedings of the 2014 new security paradigms workshop, pp 53–62

Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494CrossRef

Colom R, Flores-Mendoza C, Quiroga MÁ, Privado J (2005) Working memory and general intelligence: The role of short-term storage. Pers Indiv Diff 39(5):1005–1014CrossRef

Conway A, Jarrold C, Miyake A (2008) Variation in working memory. Oxford University PressCrossRef

Coulter NS (1983) Software science and cognitive psychology. IEEE Trans Softw Eng 2:166–171CrossRef

Crawford JR (1998) Introduction to the assessment of attention and executive functioning. Neuropsychol Rehabil 8(3):209–211CrossRef

Di Nucci D, Palomba F, De Rosa G, Bavota G, Oliveto R, De Lucia A (2017) A developer centered bug prediction model. IEEE Trans Softw Eng 44(1):5–24CrossRef

Ebisch SJ, Mantini D, Romanelli R, Tommasi M, Perrucci MG, Romani GL, Colom R, Saggino A (2013) Long-range functional interactions of anterior insula and medial frontal cortex are differently modulated by visuospatial and inductive reasoning tasks. Neuroimage 78:426–438CrossRef

Efroymson MA (1960) Multiple regression analysis. Mathematical Methods for Digital Computers pp 191–203

Eriksen BA, Eriksen CW (1974) Effects of noise letters upon the identification of a target letter in a nonsearch task. Percept Psychophys 16(1):143–149CrossRef

Fan J, McCandliss BD, Sommer T, Raz A, Posner MI (2002) Testing the efficiency and independence of attentional networks. Journal of cognitive neuroscience 14(3):340–347CrossRef

Fellows RP, Schmitter-Edgecombe M (2020) Symbol digit modalities test: Regression-based normative data and clinical utility. Arch Clin Neuropsychol 35(1):105–115CrossRef

Fisher RA (1922) On the interpretation of \(\chi \) 2 from contingency tables, and the calculation of p. J R Stat Soc 85(1):87–94CrossRef

Floyd B, Santander T, Weimer W (2017) Decoding the representation of code in the brain: An fmri study of code review and expertise. In: 2017 IEEE/ACM 39th International conference on software engineering (ICSE), IEEE, pp 175–186

Forn C, Belloch V, Bustamante JC, Garbin G, Parcet-Ibars MÀ, Sanjuan A, Ventura N, Ávila C (2009) A symbol digit modalities test version suitable for functional mri studies. Neurosci Lett 456(1):11--14CrossRef

Forn C, Rocca MA, Boscá I, Casanova B, Sanjuan A, Filippi M (2013) Analysis of “task-positive” and “task-negative” functional networks during the performance of the symbol digit modalities test in patients at presentation with clinically isolated syndrome suggestive of multiple sclerosis. Exp Brain Res 225(3):399–407

Friedman NP, Miyake A (2004) The relations among inhibition and interference control functions: a latent-variable analysis. J Exp Psychol: General 133(1):101CrossRef

Gellman MD, Turner JR (eds) (2013) Cognition, Springer New York, pp 441–441. https://doi.org/10.1007/978-1-4419-1005-9_100314

Genova HM, Hillary FG, Wylie G, Rypma B, Deluca J (2009) Examination of processing speed deficits in multiple sclerosis using functional magnetic resonance imaging. J Int Neuropsychol Soc 15(3):383–393CrossRef

Gershon RC, Wagster MV, Hendrie HC, Fox NA, Cook KF, Nowinski CJ (2013) Nih toolbox for assessment of neurological and behavioral function. Neurology 80(11 Supplement 3):S2–S6

Green CS, Bavelier D (2007) Action-video-game experience alters the spatial resolution of vision. Psychol Sci 18(1):88–94CrossRef

Habib M, Besson M (2009) What do music training and musical experience teach us about brain plasticity? Music Percept 26(3):279–285CrossRef

Halberda J, Mazzocco MM, Feigenson L (2008) Individual differences in non-verbal number acuity correlate with maths achievement. Nature 455(7213):665–668CrossRef

He Z, Shu F, Yang Y, Li M, Wang Q (2012) An investigation on the feasibility of cross-project defect prediction. Autom Softw Eng 19(2):167–199CrossRef

Howard SJ, Johnson J, Pascual-Leone J (2014) Clarifying inhibitory control: Diversity and development of attentional inhibition. Cogn Dev 31:1–21CrossRef

Huang Y, Liu X, Krueger R, Santander T, Hu X, Leach K, Weimer W (2019) Distilling neural representations of data structure manipulation using fmri and fnirs. In: 2019 IEEE/ACM 41st international conference on software engineering (ICSE), IEEE, pp 396–407

Hughes DJ (2018) Psychometric validity: Establishing the accuracy and appropriateness of psychometric measures. A multidisciplinary reference on survey, scale and test development, The Wiley handbook of psychometric testing, pp 751–779

Hutchins M, Foster H, Goradia T, Ostrand T (1994) Experiments on the effectiveness of dataflow-and control-flow-based test adequacy criteria. In: Proceedings of 16th international conference on software engineering, IEEE, pp 191–200

Juristo N, Moreno AM (2013) Basics of software engineering experimentation. Springer Science & Business Media

Karas Z, Jahn A, Weimer W, Huang Y (2021) Connecting the dots: rethinking the relationship between code and prose writing with functional connectivity. In: Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 767–779

Koen JD, Aly M, Wang WC, Yonelinas AP (2013) Examining the causes of memory strength variability: Recollection, attention failure, or encoding variability? J Exp Psychol Learn Mem Cogn 39(6):1726CrossRef

Krueger R, Huang Y, Liu X, Santander T, Weimer W, Leach K (2020) Neurological divide: an fmri study of prose and code writing. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), IEEE, pp 678–690

Kuutila M, Mäntylä MV, Claes M, Elovainio M (2017) Reviewing literature on time pressure in software engineering and related professions: computer assisted interdisciplinary literature review. In: 2017 IEEE/ACM 2nd international workshop on emotion awareness in software engineering (SEmotion), IEEE, pp 54–59

Lachman ME, Agrigoroaei S, Tun PA, Weaver SL (2014) Monitoring cognitive functioning: psychometric properties of the brief test of adult cognition by telephone. Assessment 21(4):404–417CrossRef

LeetCode (2020a) Leetcode problem: Duplicate zeros. https://leetcode.com/problems/duplicate-zeros/

LeetCode (2020b) Leetcode problem: Find and replace pattern. https://leetcode.com/problems/find-and-replace-pattern/

LeetCode (2020c) Leetcode problem: Regular expression matching. https://leetcode.com/problems/regular-expression-matching/

LeetCode (2020d) Leetcode problem: Sort the matrix diagonally. https://leetcode.com/problems/sort-the-matrix-diagonally/

Lesage E, Sutherland MT, Ross TJ, Salmeron BJ, Stein EA (2020) Nicotine dependence (trait) and acute nicotinic stimulation (state) modulate attention but not inhibitory control: converging fmri evidence from go-nogo and flanker tasks. Neuropsychopharmacology 45(5):857--865CrossRef

Li Z, Jing XY, Zhu X (2018) Progress on approaches to software defect prediction. Iet Software 12(3):161–175CrossRef

Lilienthal L, Tamez E, Shelton JT, Myerson J, Hale S (2013) Dual n-back training increases the capacity of the focus of attention. Psychon Bull Rev 20(1):135–141

Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics pp 50–60

Martínez K, Burgaleta M, Román FJ, Escorial S, Shih PC, Quiroga MÁ, Colom R (2011) Can fluid intelligence be reduced to ’simple’short-term storage? Intelligence 39(6):473–480CrossRef

Matchock RL, Mordkoff JT (2009) Chronotype and time-of-day influences on the alerting, orienting, and executive components of attention. Exp brain Res 192(2):189–198CrossRef

Mundy E, Gilmore CK (2009) Children’s mapping between symbolic and nonsymbolic representations of number. J Exp Child Psycholo 103(4):490–502CrossRef

Musso M, Kyndt E, Cascallar E, Dochy F (2012) Predicting mathematical performance: The effect of cognitive processes and self-regulation factors. Education Research International 2012

Nour S, Struys E, Stengers H (2019) Attention network in interpreters: The role of training and experience. Behav Sci 9(4):43CrossRef

Oded Y (2011) Biofeedback-based mental training in the military–the “mental gym™” project. Biofeedback 39(3):112–118

Oliveira D, Rosenthal M, Morin N, Yeh KC, Cappos J, Zhuang Y (2014) It’s the psychology stupid: how heuristics explain software vulnerabilities and how priming can illuminate developer’s blind spots. In: Proceedings of the 30th annual computer security applications conference, pp 296–305

Oliveira DS, Lin T, Rahman MS, Akefirad R, Ellis D, Perez E, Bobhate R, DeLong LA, Cappos J, Brun Y (2018) API blindspots: Why experienced developers write vulnerable code. In: Fourteenth symposium on usable privacy and security (\(\{\)SOUPS\(\}\) 2018), pp 315–328

Parmenter B, Weinstock-Guttman B, Garg N, Munschauer F, Benedict RH (2007) Screening for cognitive impairment in multiple sclerosis using the symbol digit modalities test. Multiple Sclerosis J 13(1):52–57CrossRef

Passolunghi MC, Vercelloni B, Schadee H (2007) The precursors of mathematics learning: Working memory, phonological ability and numerical competence. Cogn Dev 22(2):165–184CrossRef

Peitek N, Siegmund J, Apel S, Kästner C, Parnin C, Bethmann A, Leich T, Saake G, Brechmann A (2018) A look into programmers’ heads. IEEE Trans Softw Eng 46(4):442–462CrossRef

Piantadosi V, Scalabrino S, Serebrenik A, Novielli N, Oliveto R (2021) Replication package of "do attention and memory explain the performance of software developers?". https://figshare.com/s/f3cf009d98ac60530ec6

Pinto G, Rebouças M, Castor F (2017) Inadequate testing, time pressure, and (over) confidence: a tale of continuous integration users. In: 2017 IEEE/ACM 10th international workshop on cooperative and human aspects of software engineering (CHASE), IEEE, pp 74–77

Posner MI (1980) Orienting of attention. Q J Exp Psychol 32(1):3–25CrossRef

Posner MI, Petersen SE (1990) The attention system of the human brain. Ann Rev Neurosci 13(1):25–42CrossRef

Posnett D, D’Souza R, Devanbu P, Filkov V (2013) Dual ecological measures of focus in software development. In: 2013 35th international conference on software engineering (ICSE), IEEE, pp 452–461

Rahman F, Posnett D, Devanbu P (2012) Recalling the" imprecision" of cross-project defect prediction. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering, pp 1–11

Rapport MD, Kofler MJ, Alderson RM, Timko TM Jr, DuPaul GJ (2009) Variability of attention processes in adhd: Observations from the classroom. J Atten Disord 12(6):563–573CrossRef

Rasch RH, Tosi HL (1992) Factors affecting software developers’ performance: An integrated approach. MIS quarterly pp 395–413

Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2007) The role of experience and ability in comprehension tasks supported by uml stereotypes. In: 29th International conference on software engineering (ICSE’07), IEEE, pp 375–384

Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2009) How developers’ experience and ability influence web application comprehension tasks supported by uml stereotypes: A series of four experiments. IEEE Trans Softw Eng 36(1):96–118CrossRef

Roy E (2013) Cognitive Function, Springer New York, New York, NY, pp 448–449. https://doi.org/10.1007/978-1-4419-1005-9_1117

Scalabrino S, Linares-Vásquez M, Oliveto R, Poshyvanyk D (2018) A comprehensive model for code readability. J Softw: Evol Process 30(6):e1958

Scalabrino S, Bavota G, Vendome C, Poshyvanyk D, Oliveto R, et al (2019) Automatically assessing code understandability. IEEE Trans Softw Eng

Schacter DL, Wagner AD, Buckner RL (2000) Memory systems of 1999. Oxford University Press

Schellenberg EG (2004) Music lessons enhance iq. Psychol Sci 15(8):511–514CrossRef

Sharafi Z, Huang Y, Leach K, Weimer W (2021) Toward an objective measure of developers’ cognitive activities. ACM Trans Softw Eng Methodol (TOSEM) 30(3):1–40CrossRef

Shneiderman B, Mayer R (1979) Syntactic/semantic interactions in programmer behavior: A model and experimental results. Int J Comput Sci Inf Sci 8(3):219–238MATHCrossRef

Shorey C, Friedman E (2018) Multimorbidity and cognitive decline in a national sample of aging adults. Innovation in Aging 2(Suppl 1):505CrossRef

Siegmund J, Kästner C, Liebig J, Apel S, Hanenberg S (2014) Measuring and modeling programming experience. Empir Softw Eng 19(5):1299–1334CrossRef

Siegmund J, Peitek N, Parnin C, Apel S, Hofmeister J, Kästner C, Begel A, Bethmann A, Brechmann A (2017) Measuring neural efficiency of program comprehension. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, pp 140–150

Silva P, Spedo C, Barreira AA, Leoni RF (2018) Symbol digit modalities test adaptation for magnetic resonance imaging environment: A systematic review and meta-analysis. Mult. Sclerand Relat Disord 20:136–143CrossRef

Sloan L (2015) Learn about Spearman’s Rank-order Correlation Coefficient in SPSS with Data from the General Social Survey (2012). SAGE Publications

Song MK, Ward SE, Bair E, Weiner LJ, Bridgman JC, Hladik GA, Gilet CA (2015) Patient-reported cognitive functioning and daily functioning in chronic dialysis patients. Hemodial Int 19(1):90–99CrossRef

Spearman C (1961) The proof and measurement of association between two things. Am J Psychol

Thota MK, Shajin FH, Rajesh P et al (2020) Survey on software defect prediction techniques. International Journal of Applied Science and Engineering 17(4):331–344

Tiego J, Testa R, Bellgrove MA, Pantelis C, Whittle S (2018) A hierarchical model of inhibitory control. Front Psychol 9:1339CrossRef

Tun PA, Lachman ME (2006) Telephone assessment of cognitive function in adulthood: the brief test of adult cognition by telephone. Age and Ageing 35(6):629–632CrossRef

Wang H, Fan J, Yang Y (2004) Toward a multilevel analysis of human attentional networks. In: Proceedings of the annual meeting of the cognitive science society, vol 26

Weaver B, Bedard M, McAuliffe J, Parkkari M (2009) Using the attention network test to predict driving test scores. Accid Anal Prev 41(1):76–83CrossRef

Weaver B, Bédard M, McAuliffe J (2013) Evaluation of a 10-minute version of the attention network test. The Clinical Neuropsychologist 27(8):1281–1299CrossRef

Wei W, Yuan H, Chen C, Zhou X (2012) Cognitive correlates of performance in advanced mathematics. Br J Educ Psychol 82(1):157–181CrossRef

Wong WE, Horgan JR, London S, Mathur AP (1998) Effect of test set minimization on fault detection effectiveness. Softw: Pract Experience 28(4):347–369

Wong WE, Debroy V, Gao R, Li Y (2013) The dstar method for effective software fault localization. IEEE Trans Reliab 63(1):290–308CrossRef

Woumans E, Ceuleers E, Van der Linden L, Szmalec A, Duyck W (2015) Verbal and nonverbal cognitive control in bilinguals and interpreters. J Exp Psychol: Learn Mem Cogn 41(5):1579

Zelazo PD, Anderson JE, Richler J, Wallner-Allen K, Beaumont JL, Weintraub S (2013) Ii. nih toolbox cognition battery (cb): Measuring executive function and attention. Monogr Soc Res Child Dev 78(4):16–33

Zhang H, Gong L, Versteeg S (2013) Predicting bug-fixing time: an empirical study of commercial software projects. In: 2013 35th international conference on software engineering (ICSE), IEEE, pp 1042–1051

Zimmermann T, Nagappan N, Gall H, Giger E, Murphy B (2009) Cross-project defect prediction: a large scale experiment on data vs. domain vs. process. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, pp 91–100

Title: Do attention and memory explain the performance of software developers?
Authors: Valentina Piantadosi
Simone Scalabrino
Alexander Serebrenik
Nicole Novielli
Rocco Oliveto
Publication date: 01-09-2023
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 5/2023
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-023-10316-9

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 5/2023

More than React: Investigating the Role of Emoji Reaction in GitHub Pull Requests

What constitutes debugging? An exploratory study of debugging episodes

Tag that issue: applying API-domain labels in issue tracking systems

Responding to change over time: A longitudinal case study on changes in coordination mechanisms in large-scale agile

Using the uniqueness of global identifiers to determine the provenance of Python software source code

What have we learned? A conceptual framework on New Zealand software professionals and companies’ response to COVID-19

Premium Partner