Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Bücher
Zeitschriften
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
SLX-Digitalkonferenz: Zukunftswerkstatt 2024

Mehr Umsatz mit Top-Kontakten

Am 7. Mai erfahren Sie, wie Vertriebsteams Leadgenerierung, Leadnurturing und Leadstrategien effizient steuern können.
Erweiterte Suche
Anmelden
nach oben
Empirical Software Engineering
Erschienen in: Empirical Software Engineering 4/2016

01.08.2016

Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts

verfasst von: Leif Jonsson, Markus Borg, David Broman, Kristian Sandahl, Sigrid Eldh, Per Runeson

Erschienen in: Empirical Software Engineering | Ausgabe 4/2016

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Bug report assignment is an important part of software maintenance. In particular, incorrect assignments of bug reports to development teams can be very expensive in large software development projects. Several studies propose automating bug assignment techniques using machine learning in open source software contexts, but no study exists for large-scale proprietary projects in industry. The goal of this study is to evaluate automated bug assignment techniques that are based on machine learning classification. In particular, we study the state-of-the-art ensemble learner Stacked Generalization (SG) that combines several classifiers. We collect more than 50,000 bug reports from five development projects from two companies in different domains. We implement automated bug assignment and evaluate the performance in a set of controlled experiments. We show that SG scales to large scale industrial application and that it outperforms the use of individual classifiers for bug assignment, reaching prediction accuracies from 50 % to 89 % when large training sets are used. In addition, we show how old training data can decrease the prediction accuracy of bug assignment. We advice industry to use SG for bug assignment in proprietary contexts, using at least 2,000 bug reports for training. Finally, we highlight the importance of not solely relying on results from cross-validation when evaluating automated bug assignment.
Vorheriger Artikel Archetypal personalities of software engineers and their work preferences: a new perspective for empirical studies
Nächster Artikel Foreword to the special issue on empirical evidence on software product line engineering

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren
Fußnoten
1
Other common names for bug report include issues, tickets, fault reports, trouble reports, defect reports, anomaly reports, maintenance requests, and incidents.
 
2
Formerly Ohloh.net, an open public library presenting analyses of OSS projects (www.​openhub.​net).
 
3
Equivalent to recall when recommending only the most probable developer, aka. the Top-1 recommendation or Rc@1.
 
4
Ten levels of automation, ranging from 0, for fully manual work, to 10, when the computer acts autonomously ignoring the human.
 
5
Functional safety - Safety instrumented systems for the process industry sector.
 
6
Functional safety of Electrical/Electronic/Programmable Electronic safety-related systems.
 
7
Due to confidentiality reasons these numbers are not broken down in exact detail per project.
 
8
Term Frequency-Inverse Document Frequency (TF-IDF) is a standard weighting scheme for information retrieval and text mining. This scheme is common in software engineering applications (Borg et al. 2014).
 
Literatur
Aberdour M (2007) Achieving quality in open-source software. IEEE Softw 24 (1):58–64CrossRef
Ahsan S, Ferzund J, Wotawa F (2009) Automatic software bug triage system (bts) based on latent semantic indexing and support vector machine. In: Proceedings of the 4th international conference on software engineering advances, pp 216–221
Alenezi M, Magel K, Banitaan S (2013) Efficient bug triaging using text mining. J Softw 8(9)
Alshammari R, Zincir-Heywood A (2009) Machine learning based encrypted traffic classification: Identifying SSH and Skype. In: Proceedings of the symposium on computational intelligence for security and defense applications, pp 1–8
Amamra A, Talhi C, Robert JM, Hamiche M (2012) Enhancing smartphone malware detection performance by applying machine learning hybrid classifiers. In: Kim Th, Ramos C, Kim Hk, Kiumi A, Mohammed S, Slezak D (eds) Computer applications for software engineering, disaster recovery, and business continuity, no. 340 in communications in computer and information science. Springer, Berlin, pp 131–137
Anvik J (2007) Assisting bug report triage through recommendation. Thesis, University of British Columbia
Anvik J, Murphy GC (2011) Reducing the effort of bug report triage: recommenders for development-oriented decisions. Trans Softw Eng Methodol 20(3):10:1–10:35
Anvik J, Hiew L, Murphy GC (2006) Who should fix this bug?. In: Proceedings of the 28th international conference on software engineering, New York, NY, USA, ’06, pp 361–370
Asklund U, Bendix L (2002) A study of configuration management in open source software projects. IEE Proceedings - Software 149(1):40–46CrossRef
Avazpour I, Pitakrat T, Grunske L, Grundy J (2014) Dimensions and metrics for evaluating recommendation systems. In: Robillard M, Maalej W, Walker R, Zimmermann T (eds) Recommendation systems in software engineering. Springer, pp 245–273
Baysal O, Godfrey M, Cohen R (2009) A bug you like: A framework for automated assignment of bugs. In: Proceedings of the 17th international conference on program comprehension, pp 297–298
Bettenburg N, Premraj R, Zimmermann T, Sunghun K (2008) Duplicate bug reports considered harmful... really?. In: Proceedings of the international conference on software maintenance, pp 337–345
Bezanson J, Karpinski S, Shah VB, Edelman A (2012) Julia: A fast dynamic language for technical computing. arXiv:1209.​5145
Bhattacharya P, Neamtiu I, Shelton CR (2012) Automated, highly-accurate, bug assignment using machine learning and tossing graphs. J Syst Softw 85(10):2275–2292CrossRef
Bifet A, Holmes G, Kirkby R, Pfahringer B, Massive online analysis (2010). J Mach Learn Res 11:1601–1604
Bishop CM (2006) Pattern recognition and machine learning. Springer, New YorkMATH
Borg M, Pfahl D (2011) Do better IR tools improve the accuracy of engineers’ traceability recovery? In: Proceedings of the international workshop on machine learning technologies in software engineering, pp 27–34
Borg M, Runeson P, Ardö A (2014) Recovering from a decade: A systematic mapping of information retrieval approaches to software traceability. Empir Softw Eng 19(6):1565–1616. doi:10.​1007/​s10664-013-9255-y CrossRef
Canfora G, Cerulo L (2006) Supporting change request assignment in open source development. In: Proceedings of the symposium on applied computing, pp 1767–1772
Chen L, Wang X, Liu C (2011) An approach to improving bug assignment with bug tossing graphs and bug similarities. J Softw 6(3)
Cubranic D, Murphy GC (2004) Automatic bug triage using text categorization. In: Proceedings of the 16th international conference on software engineering & knowledge engineering, pp 92–97
Frank E, Hall M, Trigg L, Holmes G, Witten I (2004) Data mining in bioinformatics using Weka. Bioinformatics 20(15):2479–2481CrossRef
Freund Y, Schapire RE (1995) A desicion-theoretic generalization of on-line learning and an application to boosting. In: Vitanyi P (ed) Computational learning theory, no. 904 in lecture notes in computer science. Springer, Berlin, pp 23–37
Green SB (1991) How many subjects does it take to do a regression analysis. Multivar Behav Res 26(3):499–510CrossRef
Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. SIGKDD Explor Newsl 11 (1):10–18CrossRef
Helming J, Arndt H, Hodaie Z, Koegel M, Narayan N (2011) Automatic sssignment of work items. In: Maciaszek LA, Loucopoulos P (eds) Proceedings of the international conference on evaluation of novel approaches to software engineering. Springer, Berlin, pp 236–250
Hofmann M, Klinkenberg R (2013) Data mining use cases and business analytics applications. CRC Press, Taylor & Francis Group, Boca Raton. ISBN: 1482205491, 9781482205497
Jeong G, Kim S, Zimmermann T (2009) Improving bug triage with bug tossing graphs. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, New York, NY, USA , pp 111–120
Jonsson L, Broman D, Sandahl K, Eldh S (2012) Towards automated anomaly report assignment in large complex systems using stacked generalization. In: Proceedings of the International conference on software testing, verification, and validation, pp 437–446
Just S, Premraj R, Zimmermann T (2008) Towards the next generation of bug tracking systems. In: Proceedings of the Symposium on visual languages and Human-centric computing, IEEE Computer Society, pp 82–85
Kagdi H, Gethers M, Poshyvanyk D, Hammad M (2012) Assigning change requests to software developers. J Softw: Evolution and Process 24(1):3–33
Kodovsky J (2011) On dangers of cross-validation in steganalysis. Tech. rep., Birmingham University
Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th International joint conference on artificial intelligence, vol 2 , pp 1137–1143
Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207CrossRefMATH
Li N, Li Z, Nie Y, Sun X, Li X (2011) Predicting software black-box defects using stacked generalization. In: Proceedings of the International conference on digital information management , pp 294–299
Li Q, Wang Q, Yang Y, Li M (2008) Reducing biases in individual software effort estimations: a combining approach. In: Proceedings of the 2nd international symposium on empirical software engineering and measurement. doi:10.​1145/​1414004.​1414041, pp 223–232
Lin Z, Shu F, Yang Y, Hu C, Wang Q (2009) An empirical study on bug assignment automation using Chinese bug data. In: Proceedings of the 3rd international symposium on empirical software engineering and measurement, pp 451–455
Linares-Vasquez M, Hossen K, Dang H, Kagdi H, Gethers M, Poshyvanyk D (2012) Triaging incoming change requests: bug or commit history, or code authorship?. In: Proceedings of the 28th international conference on software maintenance, pp 451–460
Matter D, Kuhn A, Nierstrasz O (2009) Assigning bug reports using a vocabulary-based expertise model of developers. In: 6th IEEE International working conference on mining software repositories, 2009. MSR ’09. doi:10.​1109/​MSR.​2009.​5069491, pp 131–140
Murphy-Hill E, Murphy G (2014) Recommendation delivery: getting the user interface just right. In: Robillard M, Maalej W, Walker R, Zimmermann T (eds) Recommendation systems in software engineering. Springer, Berlin
Nagwani N, Verma S (2012) Predicting expert developers for newly reported bugs using frequent terms similarities of bug attributes. In: Proceedings of the 9th international conference on ICT and knowledge engineering, pp 113–117
Owen S, Anil R, Dunning T, Friedman E (2011) Mahout in action. Manning Publications, Shelter Island
Parasuraman R, Sheridan T, Wickens C (2000) A model for types and levels of human interaction with automation. IEEE Trans Syst Man Cybern 30(3):286–297CrossRef
Park J, Lee M, Kim J, Hwang S, Kim S (2011) A cost-aware triage algorithm for bug reporting systems. In: Proceedings of the 25th AAAI conference on artificial intelligence
Paulson J, Succi G, Eberlein A (2004) An empirical study of open-source and closed-source software products. IEEE Trans Softw Eng 30(4):246–256CrossRef
Petersen K, Wohlin C (2009) Context in industrial software engineering research. In: Proceedings of the 3rd international symposium on empirical software engineering and measurement, pp 401–404
Rao R, Fung G, Rosales R (2008) On the dangers of cross-validation. An experimental evaluation. In: Proceedings of the SIAM international conference on data mining, pp 588–596
Robillard M, Maalej W, Walker R, Zimmermann T (2014) Recommendation systems in software engineering. Springer, BerlinCrossRef
Robinson B, Francis P (2010) Improving industrial adoption of software engineering research: A comparison of open and closed source software. In: Proceedings of the international symposium on empirical software engineering and measurement, pp 21:1–21:10
Robles G, Gonzalez-Barahona J (2006) Contributor turnover in Libre software projects. In: Damiani E, Fitzgerald B, Scacchi W, Scotto M, Succi G (eds) Open source systems, no. 203 in International federation for information processing. Springer, pp 273–286
Servant F, Jones J (2012) Automatic developer-to-fault assignment through fault localization. In: Proceedings. of the 34th international conference on software engineering (ICSE), pp 36–46
Shokripour R, Kasirun Z, Zamani S, Anvik J (2012) Automatic bug assignment using information extraction methods. In: Proceedings of the international conference on advanced computer science applications and technologies, pp 144–149
Sill J, Takacs G, Mackey L, Lin D (2009) Feature-weighted linear stacking. arXiv:0911.​0460
Tamrawi A, Nguyen T, Al-Kofahi J, Nguyen T (2011) Fuzzy set and cache-based approach for bug triaging. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on foundations of software engineering. doi:10.​1145/​2025113.​2025163, pp 365–375
Thomas S, Nagappan M, Blostein D, Hassan A (2013) The impact of classifier configuration and classifier combination on bug localization. IEEE Trans. Softw. Eng. 39(10):1427–1443CrossRef
Tibshirani R, Walther G, Hastie T (2001) Estimating the number of clusters in a data set via the gap statistic. J R Stat Soc Ser B (Stat Methodol) 63(2):411–423MathSciNetCrossRefMATH
Wiklund K, Eldh S, Sundmark D, Lundqvist K (2013) Can we do useful industrial software engineering research in the shadow of lean and agile?. In: Proceedings of the 1st international workshop on conducting empirical studies in industry, pp 67–68
Witten IH, Frank E, Hall MA (2011) Data mining. pub. Burlington, MA
Wohlin C, Runeson P, Host M, Ohlsson M, Regnell B, Wesslen A (2012) Experimentation in software engineering: A practical guide. Springer, BerlinCrossRefMATH
Wu W, Zhang W, Yang Y, Wang Q (2011) Developer recommendation with k-nearest-neighbor search and expertise ranking. In: Proceedings of the 18th Asia pacific software engineering conference, pp 389–396
Xia X, Lo D, Wang X, Zhou B (2013) Accurate developer recommendation for bug resolution. In: Proceedings of the 20th working conference on reverse engineering, pp 72–81
Xie X, Zhang W, Yang Y, Wang Q (2012) Developer recommendation based on topic models for bug resolution. In: Proceedings of the 8th international conference on predictive models in software engineering, pp 19–28
Zaharia M, Chowdhury NMM, Franklin M, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. Tech. rep., EECS department, University of California, University of California at Berkeley, Berkeley, California
Zhao Y, Zhang Y (2008) Comparison of decision tree methods for finding active objects. Adv Space Res 41(12):1955–1959CrossRef
Metadaten
Titel
Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts
verfasst von
Leif Jonsson
Markus Borg
David Broman
Kristian Sandahl
Sigrid Eldh
Per Runeson
Publikationsdatum
01.08.2016
Verlag
Springer US
Erschienen in
Empirical Software Engineering / Ausgabe 4/2016
Print ISSN: 1382-3256
Elektronische ISSN: 1573-7616
DOI
https://doi.org/10.1007/s10664-015-9401-9

Weitere Artikel der Ausgabe 4/2016

Empirical Software Engineering 4/2016 Zur Ausgabe

OriginalPaper

Quantifying structural attributes of system decompositions in 28 feature-oriented software product lines

OriginalPaper

Performance variability in software product lines: proposing theories from a case study

Experience Report

A systematic test case selection methodology for product lines: results and insights from an industrial case study

OriginalPaper

Using text clustering to predict defect resolution time: a conceptual replication and an evaluation of prediction accuracy

OriginalPaper

The effects of visualization and interaction techniques on feature model configuration

Editorial

Foreword to the special issue on empirical evidence on software product line engineering