nach oben

Empirical Software Engineering

Erschienen in:

01.07.2023

Come for syntax, stay for speed, understand defects: an empirical study of defects in Julia programs

verfasst von: Akond Rahman, Dibyendu Brinto Bose, Raunak Shakya, Rahul Pandita

Erschienen in: Empirical Software Engineering | Ausgabe 4/2023

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Julia has emerged as a popular programming language to develop scientific software, in part due to its flexible syntax akin to scripting languages while retaining the execution speed of a compiled language. Similar to any programming language, Julia programs are susceptible to defects. However, a systematic characterization of defects in Julia programs remains under-explored. A systematic analysis of defects in Julia programs will act as a starting point for researchers and toolsmiths in building developer tools to improve the quality of Julia programs. To this end, we conduct an empirical study with 742 defects that appear in Julia programs by mining 30,494 commits and 3,038 issue reports collected from 112 open-source Julia projects. From our empirical analysis, we identify 9 defect categories and 7 defect symptoms. We observe certain defect categories to be Julia-specific, e.g., type instability and world age defects. We also survey 52 developers to rank the identified categories based on perceived severity. Based on our empirical analysis, we provide specific recommendations for researchers and toolsmiths.

Vorheriger Artikel Code reviews in open source projects : how do gender biases affect participation and outcomes?

Nächster Artikel Refactoring with domain-driven design in an industrial context

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

https://github.com/JuliaLang/julia

https://github.com/JuliaLang/Pkg.jl

https://github.com/JuliaInterop/Cxx.jl

https://www.amazon.com/gift-cards

https://github.com/JunoLab/Atom.jl/commit/4650266b3c2a74d2f6db61e8710ae5b2cd8593a9

https://github.com/JuliaParallel/Hwloc.jl/commit/cdcf861c5ab5fa9de876344c7e2271ec69e9b916

https://github.com/JuliaParallel/Hwloc.jl/commit/65781b7c08ee0349ccf7db6c629730beb87e8625

https://github.com/JunoLab/Atom.jl/commit/080609b34d4c06dfb13950f9cdd375495d25c7f5

Berkeley CA (2017) Julia joins petaflop club. https://www.hpcwire.com/off-the-wire/julia-joins-petaflop-club/

Perkel JM (2019) Julia: come for the syntax, stay for the speed. https://www.nature.com/articles/d41586-019-02310-3

Berkeley CA (2020a) Bug types in Julia: what types of bugs in Julia programs have you experienced? https://discourse.julialang.org/t/bug-types-in-julia-what-types-of-bugs-in-julia-programs-have-you-experienced-are-they-different-from-bug-types-in-general-programming-languages-like-c-c/38640

Berkeley CA (2020b) Programming languages: developers reveal what they love and loathe, and what pays best. https://www.zdnet.com/article/programming-languages-developers-reveal-what-they-love-and-loathe-and-what-pays-best/

Berkeley CA (2020c) Why julia is slowly replacing python in machine learning and data science. https://www.section.io/engineering-education/why-julia-is-slowly-replacing-python-for-machine-learning-and-data-science/

Perkel JM (2022a) Julia. https://juliacomputing.com/case-studies/celeste.html

Perkel JM (2022b) Julia. https://juliacomputing.com/case-studies/lanl/

Perkel JM (2022c) Julia. https://juliacomputing.com/case-studies/lincoln-labs/

Perkel JM (2022d) The Julia language. https://docs.julialang.org/en/v1/

Berkeley CA (2022) The julia programming language. https://julialang.org/#

abelsiqueira (2015) https://gist.github.com/abelsiqueira/d4ca585c62204516bf37. Accessed 19 Aug 2021

Agrawal A, Rahman A, Krishna R, Sobran A, Menzies T (2018) We don’t need another hero?: The impact of “heroes” on software development. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, ICSE-SEIP ’18, pp. 245–253. ACM, New York. https://doi.org/10.1145/3183519.3183549

amckay (2016) Bug fix in markov chain simulation. https://github.com/QuantEcon/QuantEcon.jl/commit/097cf3. Accessed 17 Feb 2022

Antinyan V, Staron M, Sandberg A (2017) Evaluating code complexity triggers, use of complexity measures and the influence of code complexity on maintenance time. Empir Softw Eng 22(6):3057–3087CrossRef

Beizer B (1984) Software system testing and quality assurance. Van Nostrand Reinhold Co

Belyakova J, Chung B, Gelinas J, Nash J, Tate R, Vitek J (2020) World age in julia: Optimizing method dispatch in the presence of eval. Proc ACM Program Lang 4(OOPSLA), https://doi.org/10.1145/3428275

Bezanson J, Chen J, Chung B, Karpinski S, Shah VB, Vitek J, Zoubritzky L (2018) Julia: dynamism and performance reconciled by design. Proc ACM Program Lang 2(OOPSLA), https://doi.org/10.1145/3276490

Carver JC (2009) First international workshop on software engineering for computational science & engineering. Comput Sci Eng 11(2):7–11. https://doi.org/10.1109/MCSE.2009.30CrossRef

Catolino G, Palomba F, Zaidman A, Ferrucci F (2019) Not all bugs are the same: understanding, characterizing, and classifying bug types. J Syst Softw 152:165–181. DOIurlhttps://doi.org/10.1016/j.jss.2019.03.002, urlwww.sciencedirect.com/science/article/pii/S0164121219300536

Chen Z, Yao H, Lou Y, Cao Y, Liu Y, Wang H, Liu X (2021) An empirical study on deployment faults of deep learning based mobile applications. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE), IEEE, pp. 674–685

Chillarege R, Bhandari I, Chaar J, Halliday M, Moebus D, Ray B, Wong MY (1992) Orthogonal defect classification-a concept for in-process measurements. IEEE Trans. Softw. Eng. 18(11):943–956. https://doi.org/10.1109/32.177364CrossRef

Chillarege R, Bhandari I, Chaar J, Halliday M, Moebus D, Ray B, Wong MY (1992) Orthogonal defect classification-a concept for in-process measurements. IEEE Trans Softw Eng 18(11):943–956. https://doi.org/10.1109/32.177364CrossRef

Churavy VVR (2019) Transparent distributed programming in Julia. PhD thesis, Massachusetts Institute of Technology, https://dspace.mit.edu/handle/1721.1/122755

Cinque M, Cotroneo D, Corte RD, Pecchia A (2014) Assessing direct monitoring techniques to analyze failures of critical industrial systems. In: 2014 IEEE 25th international symposium on software reliability engineering, pp. 212–222, https://doi.org/10.1109/ISSRE.2014.30

Devanbu P, Zimmermann T, Bird C (2016) Belief and evidence in empirical software engineering. In: Proceedings of the 38th international conference on software engineering, ACM, New York, NY, USA, ICSE ’16, pp 108–119, https://doi.org/10.1145/2884781.2884812

Di Franco A, Guo H, Rubio-González C (2017) A comprehensive study of real-world numerical bug characteristics. In: 2017 32nd IEEE/ACM international conference on automated software engineering (ASE), IEEE, pp. 509–519

Drvi (2020) Fix inequality in reservoir sampling. https://github.com/joshday/OnlineStats.jl/commit/023df7. Accessed 16 Feb 2022

Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. Springer, London, London, pp 285–311

eulerkochy (2020) Fix parametric initialisation of rbtree. https://github.com/JuliaCollections/DataStructures.jl/commit/b04a52. Accessed 10 Feb 2022

Friess E (2019) Scrum language use in a software engineering firm: an exploratory study. IEEE Trans Prof Commun 62(2):130–147. https://doi.org/10.1109/TPC.2019.2911461CrossRef

Gagniuc PA (2017) Markov chains: from theory to implementation and experimentation. WileyCrossRefMATH

Garcia J, Feng Y, Shen J, Sumaya Xia AY, Chen QA (2020) A comprehensive study of autonomous vehicle bugs. In: Proceedings of the 42nd international conference on software engineering, ICSE ’20, to appear

Gibson J (2017) The julia programming language: the future of scientific computing. APS pp L39–011, https://ui.adsabs.harvard.edu/abs/2017APS..DFDL39011G/abstract

github (2023) Blocking command line pushes that expose your personal email address. https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/, Accessed 22 Feb 2023

Gmys J, Carneiro T, Melab N, Talbi EG, Tuyttens D (2020) A comparative study of high-productivity high-performance programming languages for parallel metaheuristics. Swarm and Evolutionary Computation p. 100720, https://www.sciencedirect.com/science/article/abs/pii/S2210650220303734

Hickey G, Kipping C (1996) A multi-stage approach to the coding of data from open-ended questions. Nurse Res 4(1):81–91CrossRef

Howison J, Herbsleb JD (2011) Scientific software production: Incentives and collaboration. In: Proceedings of the ACM 2011 conference on computer supported cooperative work, CSCW ’11, Association for Computing Machinery, New York, p 513–522, https://doi.org/10.1145/1958824.1958904

Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, ICSE ’20, Association for Computing Machinery, New York, p 1110–1121, https://doi.org/10.1145/3377811.3380395

Islam MJ, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ESEC/FSE 2019, Association for Computing Machinery, New York, p 510–520, https://doi.org/10.1145/3338906.3338955

Januszek T, Pleszczyński M (2018) Comparative analysis of the efficiency of Julia language against the other classic programming languages. Silesian Journal of Pure and Applied Mathematics 8, https://yadda.icm.edu.pl/baztech/element/bwmeta1.element.baztech-c4339453-4519-4b92-a673-307638a50cb1

jayschwa (2014) Fix infinite recursion on 32-bit machines. https://github.com/JuliaGL/GLFW.jl/commit/89343b, Accessed 19 Feb 2022

Julia (2021) https://docs.julialang.org/en/v1/base/numbers/#Core.UInt64, Accessed 29 Aug 2021

julia (2021) Julia computing. https://juliacomputing.com/, Accessed 11 Oct 2021

JuliaLang (2017) https://github.com/JuliaLang/IJulia.jl/issues/573, Accessed 10 Aug 2021

JuliaLang (2021) Julialang/ijulia.jl. https://github.com/JuliaLang/IJulia.jl, Accessed 10 Aug 2021

JuliaLang/IJuliajl (2017) Pkd.build(“ijulia”) failing on arm. https://github.com/JuliaLang/IJulia.jl/issues/516, Accessed 25 Aug 2021

JuliaSmoothOptimizers (2019) Juliasmoothoptimizers/linearoperators.jl. https://github.com/JuliaSmoothOptimizers/LinearOperators.jl/issues/110, Accessed 09 Aug 2021

JuliaSmoothOptimizers/CUTEstjl (2015) Segmentation fault. https://github.com/JuliaSmoothOptimizers/CUTEst.jl/issues/45, Accessed 19 Aug 2021

Kitchenham BA, Pfleeger SL (2008) Personal Opinion Surveys, Springer London, London, pp 63–92. https://doi.org/10.1007/978-1-84800-044-5_3

Krippendorff K (2018) Content analysis: An introduction to its methodology. Sage publications

Krippendorff K, Fleiss JL (1978) Reliability of binary attribute data

Krishna R, Agrawal A, Rahman A, Sobran A, Menzies T (2018) What is the connection between issues, bugs, and enhancements?: Lessons learned from 800+ software projects. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, ICSE-SEIP ’18, ACM, New York, pp 306–315, https://doi.org/10.1145/3183519.3183548

Lattner C, Adve V (2004) Llvm: A compilation framework for lifelong program analysis & transformation. In: International symposium on code generation and optimization, 2004. CGO 2004., IEEE, pp 75–86

Lombard M, Snyder-Duch J, Bracken CC (2010) Practical resources for assessing and reporting intercoder reliability in content analysis research projects

Makhshari A, Mesbah A (2021) Iot bugs and development challenges. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE), pp 460–472, https://doi.org/10.1109/ICSE43902.2021.00051

Mauny M, Vaugon B (2014) Nullable type inference. In: OCaml 2014-The OCaml users and developers workshop

Merchant FA, Castleman KR (2005) 10.10 - computer-assisted microscopy. In: BOVIK A (ed) Handbook of Image and Video Processing (Second Edition), second edition edn, Communications, Networking and Multimedia, Academic Press, Burlington, pp 1311–XLIV, https://doi.org/10.1016/B978-012119792-6/50136-4, https://www.sciencedirect.com/science/article/pii/B9780121197926501364

MikeInnes (2017) https://github.com/JunoLab/Juno.jl/commit/bd49b4, Accessed 20 Feb 2022

MOSEK (2018) fix: three number version. https://github.com/MOSEK/Mosek.jl/commit/3e9c63dab09, Accessed 09 Aug 2021

Munaiah N, Kroh S, Cabrey C, Nagappan M (2017) Curating GitHub for engineered software projects. Empirical Software Engineering pp 1–35, https://doi.org/10.1007/s10664-017-9512-6,

Murphy J, Brady ET, Shamim SI, Rahman A (2020) A curated dataset of security defects in scientific software projects. In: Proceedings of the 7th symposium on hot topics in the science of security, HotSoS ’20, Association for Computing Machinery, New York, https://doi.org/10.1145/3384217.3384218

pfitzseb (2017) https://github.com/JunoLab/Juno.jl/commit/865068, Accessed 21 Feb 2022

Poulding S, Feldt R (2017) Automated random testing in multiple dispatch languages. In: 2017 IEEE International conference on software testing, verification and validation (ICST), pp 333–344, https://doi.org/10.1109/ICST.2017.37

Rahman A (2022) Verifiability package for paper. https://figshare.com/s/35d775572bb840ebd392, Accessed 15 Mar 2022

Rahman A, Agrawal A, Krishna R, Sobran A (2018) Characterizing the influence of continuous integration: Empirical results from 250+ open source and proprietary projects. In: Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics, SWAN 2018, ACM, New York, pp 8–14, https://doi.org/10.1145/3278142.3278149,

Rahman A, Farhana E, Parnin C, Williams L (2020) Gang of eight: A defect taxonomy for infrastructure as code scripts. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, ICSE ’20, Association for Computing Machinery, New York, p 752–764, https://doi.org/10.1145/3377811.3380409, https://akondrahman.github.io/papers/icse20_acid.pdf

randyzwitch (2015) https://github.com/johnmyleswhite/Vega.jl/commit/9e3046, Accessed 19 Feb 2022

Raulamo-Jurvanen P, Hosio S, Mäntylä MV (2019) Practitioner evaluations on software testing tools. In: Proceedings of the evaluation and assessment on software engineering, EASE ’19, Association for Computing Machinery, New York, pp 57–66, https://doi.org/10.1145/3319008.3319018

Saldaña J (2015) The coding manual for qualitative researchers. Sage

Satyanarayan A, Moritz D, Wongsuphasawat K, Heer J (2016) Vega-lite: A grammar of interactive graphics. IEEE Trans Vis Comput Graph 23(1):341–350CrossRef

Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008a) Defect categorization: Making use of a decade of widely varying historical data. In: Proceedings of the Second ACM-IEEE International symposium on empirical software engineering and measurement, ESEM ’08, Association for Computing Machinery, New York, pp 149–157, https://doi.org/10.1145/1414004.1414030

Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008b) Defect categorization: Making use of a decade of widely varying historical data. In: Proceedings of the Second ACM-IEEE International symposium on empirical software engineering and measurement, ESEM ’08, Association for Computing Machinery, New York, pp 149–157, https://doi.org/10.1145/1414004.1414030

simonster (2014) Fix type instability in read_array. https://github.com/JuliaIO/HDF5.jl/commit/ce2c44, Accessed 20 Feb 2022

Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 2013 6th International workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92, https://doi.org/10.1109/CHASE.2013.6614738

Sweeney A, Greenwood KE, Williams S, Wykes T, Rose DS (2013) Hearing the voices of service user researchers in collaborative qualitative data analysis: the case for multiple coding. Health Expect 16(4):e89–e99CrossRef

Tan L, Liu C, Li Z, Wang X, Zhou Y, Zhai C (2014) Bug characteristics in open source software. Empir Softw Eng 19:1665–1705CrossRef

tknopp (2018) fix sampling density function. https://github.com/JuliaMath/NFFT.jl/commit/fc791b, Accessed 18 Feb 2022

Vitter JS (1985) Random sampling with a reservoir. ACM Trans Math Softw 11(1):37–57. https://doi.org/10.1145/3147.3165MathSciNetCrossRefMATH

julia vscode (2022) julia-vscode/cstparser.jl. https://github.com/julia-vscode/CSTParser.jl, Accessed 22 Feb 2022

Wang P, Brown C, Jennings JA, Stolee KT (2020) An empirical study on regular expression bugs. In: Proceedings of the 17th international conference on mining software repositories, MSR ’20, Association for Computing Machinery, New York, pp 103–113, https://doi.org/10.1145/3379597.3387464

WGS-TB (2021a) Update calling_functions.jl. https://github.com/WGS-TB/MentaLiST/commit/3f59f7b, Accessed 11 Aug 2021

WGS-TB (2021b) Wgs-tb/mentalist. https://github.com/WGS-TB/MentaLiST, Accessed 11 Aug 2021

yuyichao (2016) https://github.com/JuliaPy/PyCall.jl/commit/6ff741, Accessed 17 Feb 2022

Zappa Nardelli F, Belyakova J, Pelenitsyn A, Chung B, Bezanson J, Vitek J (2018) Julia subtyping: A rational reconstruction. Proc ACM Program Lang 2(OOPSLA), https://doi.org/10.1145/3276483

Zhang Y, Chen Y, Cheung SC, Xiong Y, Zhang L (2018a) An empirical study on tensorflow program bugs. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2018, Association for Computing Machinery, New York, pp 129–140, https://doi.org/10.1145/3213846.3213866

Zhang Y, Chen Y, Cheung SC, Xiong Y, Zhang L (2018b) An empirical study on tensorflow program bugs. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2018, Association for Computing Machinery, New York, pp 129–140, https://doi.org/10.1145/3213846.3213866

Titel: Come for syntax, stay for speed, understand defects: an empirical study of defects in Julia programs
verfasst von: Akond Rahman
Dibyendu Brinto Bose
Raunak Shakya
Rahul Pandita
Publikationsdatum: 01.07.2023
Verlag: Springer US
Erschienen in: Empirical Software Engineering / Ausgabe 4/2023
Print ISSN: 1382-3256
Elektronische ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-023-10328-5

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Weitere Artikel der Ausgabe 4/2023

How do programmers fix bugs as workarounds? An empirical study on Apache projects

Evaluating pre-trained models for user feedback analysis in software engineering: a study on classification of app-reviews

Towards a taxonomy of Roxygen documentation in R packages

Seeing confusion through a new lens: on the impact of atoms of confusion on novices’ code comprehension

An empirical study of the systemic and technical migration towards microservices

Learning to Predict Code Review Completion Time In Modern Code Review

Premium Partner