Top

Empirical Software Engineering

Hint

Swipe to navigate through the articles of this issue

Published in:

01-07-2021

On using Stack Overflow comment-edit pairs to recommend code maintenance changes

Authors: Henry Tang, Sarah Nadi

Published in: Empirical Software Engineering | Issue 4/2021

Abstract

Code maintenance data sets typically consist of a before version of the code and an after version that contains the improvement or fix. Such data sets are important for various software engineering support tools related to code maintenance, such as program repair, code recommender systems, or Application Programming Interface (API) misuse detection. Most of the current data sets are typically constructed from mining commit history in version-control systems or issues in issue-tracking systems. In this paper, we investigate whether Stack Overflow can be used as an additional source for building code maintenance data sets. Comments on Stack Overflow provide an effective way for developers to point out problems with existing answers, alternative solutions, or pitfalls. Given its crowd-sourced nature, answers are then updated to incorporate these suggestions. In this paper, we mine comment-edit pairs from Stack Overflow and investigate their potential usefulness for constructing the above data sets. These comment-edit pairs have the added benefit of having concrete descriptions/explanations of why the change is needed as well as potentially having less tangled changes to deal with. We first design a technique to extract related comment-edit pairs and then qualitatively and quantitatively investigate the nature of these pairs. We find that the majority of comment-edit pairs are not tangled, but find that only 27% of the studied pairs are potentially useful for the above applications. We categorize the types of mined pairs and find that the highest ratio of useful pairs come from those categorized as Correction, Obsolete, Flaw, and Extension. These categories can provide data for both corrective and preventative maintenance activities. To demonstrate the effectiveness of our extracted pairs, we submitted 15 pull requests to popular GitHub repositories, 10 of which have been accepted to widely used repositories such as Apache Beam (https://beam.apache.org/) and NLTK (https://www.nltk.org/). Our work is the first to investigate Stack Overflow comment-edit pairs and opens the door for future work in this direction. Based on our findings and observations, we provide concrete suggestions on how to potentially identify a larger set of useful comment-edit pairs, which can also be facilitated by our shared data.

previous article Understanding and improving artifact sharing in software engineering research

next article Development of recommendation systems for software engineering: the CROSSMINER experience

To get access to this content you need the following product:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

http://stackoverflow.com/questions/50383046

http://stackoverflow.com/questions/19694159

http://stackoverflow.com/questions/24261462

http://stackoverflow.com/questions/52517618

Note that we use the term code recommender system as a general umbrella for any support tool that suggests fixes, code changes, or related code snippets.

https://beam.apache.org/

https://www.nltk.org/

note that they refer to this message as comment in their paper, but it is not a comment on the answer, but rather the message the editor provides with their edit

http://stackoverflow.com/questions/281433

https://stackoverflow.com/revisions/281433/3

http://stackoverflow.com/questions/52347606

http://stackoverflow.com/questions/62680

http://stackoverflow.com/questions/34459380

https://stackoverflow.com/revisions/34459380/3

http://stackoverflow.com/questions/44765572

https://stackoverflow.com/revisions/44765572/9

https://stackoverflow.com/questions/6872517

https://stackoverflow.com/revisions/6872517/2

https://stackoverflow.com/questions/5616616

https://stackoverflow.com/revisions/5616616/5

https://stackoverflow.com/revisions/5169321/6

https://stackoverflow.com/questions/5169321

https://github.com/ualberta-smr/QueryGitHub

https://stackoverflow.com/questions/52517618

https://stackoverflow.com/revisions/52517618/4

https://github.com/apache/beam/pull/11017

https://stackoverflow.com/questions/4605982

https://beam.apache.org/

https://www.nltk.org/

https://stackoverflow.com/questions/26933338

https://stackoverflow.com/questions/53216022

https://stackoverflow.com/questions/44470955

https://stackoverflow.com/questions/20051167

Adaji I, Vassileva J (2016) Modelling user collaboration in social networks using edits and comments. In: Proceedings of the 2016 conference on user modeling adaptation and personalization, ser. UMAP ’16. ACM, New York, pp 111–114. [Online]. Available: https://doi.org/10.1145/2930238.2930289

Amann S, Nadi S, Nguyen HA, Nguyen TN, Mezini M (2016) MUBench: A benchmark for API-misuse detectors. In: 2016 IEEE/ACM 13th working conference on mining software repositories. IEEE, pp 464–467

Amann S, Nguyen HA, Nadi S, Nguyen TN, Mezini M (2018) A systematic evaluation of static API-misuse detectors. IEEE Trans Softw Eng

Bachmann A, Bird C, Rahman F, Devanbu P, Bernstein A (2010) The missing links: Bugs and bug-fix commits. In: Proceedings of the Eighteenth ACM SIGSOFT international symposium on foundations of software engineering, ser. FSE ’10. ACM, New York, pp 97–106. [Online]. Available: https://doi.org/10.1145/1882291.1882308

Baltes S (2018) Edit and comment history of Stack Overflow threads. [Online]. Available: https://empirical-software.engineering/blog/sotorrent-edithistory

Baltes S, Treude C, Diehl S (2018) SOTorrent: Studying the origin, evolution, and usage of Stack Overflow code snippets. arXiv: 1809.02814. [Online]. Available: 1809.02814

Barua A, Thomas SW, Hassan AE (2014) What are developers talking about? An analysis of topics and trends in Stack Overflow. Empir Softw Eng 19 (3):619–654. [Online]. Available: https://doi.org/10.1007/s10664-012-9231-y CrossRef

Bird C, Bachmann A, Aune E, Duffy J, Bernstein A, Filkov V, Devanbu P (2009) Fair and balanced?: Bias in bug-fix datasets. In: Proceedings of the 7th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, ser. ESEC/FSE ’09. ACM, New York, pp 121–130. [Online]. Available: https://doi.org/10.1145/1595696.1595716

Bissyandé TF, Thung F, Wang S, Lo D, Jiang L, Ré veillère L (2013) Empirical evaluation of bug linking. In: 2013 17th European conference on software maintenance and reengineering, pp 89–98

ChairNerd (2011) Fuzzywuzzy: Fuzzy string matching in Python. [Online]. Available: https://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/

Chan SWK, Franklin J (2003) Dynamic context generation for natural language understanding: A multifaceted knowledge approach. IEEE Trans Sys Man Cybern Part A Syst Hum 33(1):23–41 CrossRef

Cifuentes C, Hoermann C, Keynes N, Li L, Long S, Mealy E, Mounteney M, Scholz B (2009) BegBunch: Benchmarking for C bug detection tools. In: Proceedings of the 2nd international workshop on defects in large software systems: held in conjunction with the ACM SIGSOFT international symposium on software testing and analysis, pp 16–20

Cortés-Coy LF, Linares-Vásquez M, Aponte J, Poshyvanyk D (2014) On automatically generating commit messages via summarization of source code changes. In: 14th IEEE international working conference on source code analysis and manipulation, pp 275–284

Dalip DH, Gonçalves MA, Cristo M, Calado P (2013) Exploiting user feedback to learn to rank answers in Q&A forums: A case study with Stack Overflow. In: Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval, ser. SIGIR ’13. ACM, New York. [Online]. Available: https://doi.org/10.1145/2484028.2484072

Dallmeier V, Zimmermann T (2007) Extraction of bug localization benchmarks from history. In: Proceedings of the Twenty-second IEEE/ACM international conference on automated software engineering, ser. ASE ’07. ACM, New York, pp 433–436. [Online]. Available: https://doi.org/10.1145/1321631.1321702

Dallmeier V, Zimmermann T (2007) Extraction of bug localization benchmarks from history. In: Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, pp 433–436

Diamantopoulos T, Sifaki M-I, Symeonidis AL (2019) Towards mining answer edits to extract evolution patterns in Stack Overflow. In: Proceedings of the 16th international conference on mining software repositories, ser. MSR ’19. IEEE Press, Piscataway, pp 215–219. [Online]. Available: https://doi.org/10.1109/MSR.2019.00043

Dit B, Holtzhauer A, Poshyvanyk D, Kagdi H (2013) A dataset from change history to support evaluation of software maintenance tasks. In: 2013 10th working conference on mining software repositories. IEEE, pp 131–134

Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 international conference on software engineering. IEEE Press, pp 422–431

Falleri J, Morandat F, Blanc X, Martinez M, Monperrus M (2014) Fine-grained and accurate source code differencing. In: ACM/IEEE international conference on automated software engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014, pp 313–324. [Online]. Available: https://doi.org/10.1145/2642937.2642982

Fenton NE, Neil M (1999) A critique of software defect prediction models. IEEE Trans Softw Eng 25(5):675–689 CrossRef

Ferenc R, Gyimesi P, Gyimesi G, Tóth Z, Gyimóthy T (2020) An automatically created novel bug dataset and its validation in bug prediction. J Sys Softw 169:110691. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0164121220301436 CrossRef

Ganguly D, Roy D, Mitra M, Jones GJ (2015) Word embedding based generalized language model for information retrieval. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp 795–798

Gao Q, Zhang H, Wang J, Xiong Y, Zhang L, Mei H (2015) Fixing recurring crash bugs via analyzing Q&A sites (t). In: 2015 30th IEEE/ACM international conference on automated software engineering. IEEE, pp 307–318

Gazzola L, Micucci D, Mariani L (2019) Automatic software repair: a survey. IEEE Trans Softw Eng 45(1):34–67 CrossRef

Herzig K, Just S, Zeller A (2013) It’s not a bug, it’s a feature: How misclassification impacts bug prediction. In: Proceedings of the 2013 international conference on software engineering, ser. ICSE ’13. IEEE Press, Piscataway, pp 392–401. [Online]. Available: http://dl.acm.org/citation.cfm?id=2486788.2486840

Herzig K, Zeller A (2013) The impact of tangled code changes. In: 2013 10th working conference on mining software repositories. IEEE, pp 121–130

Herzig K, Just S, Zeller A (2016) The impact of tangled code changes on defect prediction models. Empir Softw Eng 21(2):303–336. [Online]. Available: https://doi.org/10.1007/s10664-015-9376-6 CrossRef

Horton E, Parnin C (2019) Dockerizeme:, Automatic inference of environment dependencies for Python code snippets. In: 41st IEEE/ACM international conference on software engineering, pp 328–338

Jin X, Servant F (2019) What edits are done on the highly answered questions in Stack Overflow?: An empirical study. In: Proceedings of the 16th international conference on mining software repositories, ser. MSR ’19. IEEE Press, Piscataway, pp 225–229. [Online]. Available: https://doi.org/10.1109/MSR.2019.00045

Just R, Jalali D, Ernst MD (2014) Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In: Proceedings of the 2014 international symposium on software testing and analysis, ser. ISSTA ’14. ACM, New York, pp 437–440. [Online]. Available: https://doi.org/10.1145/2610384.2628055

Kim S, Zimmermann T, Pan K, Whitehead EJ Jr (2006) Automatic identification of bug-introducing changes. In: 21st IEEE/ACM international conference on automated software engineering, pp 81–90

Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions and reversals. Soviet Phys Doklady 110(18):707–710. doklady Akademii Nauk SSSR, V163 No4, 845–848 (1965) MathSciNet

Lin B, Zampetti F, Bavota G, Di Penta M, Lanza M (2019) Pattern-based mining of opinions in Q&A websites. In: Proceedings of the 41st international conference on software engineering, ser. ICSE ’19. IEEE Press, Piscataway, pp 548–559. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00066

Liu C, Yang J, Tan L, Hafiz M (2013) r2fix: Automatically generating bug fixes from bug reports. In: 2013 IEEE Sixth international conference on software testing, verification and validation, pp 282–291

Liu X, Zhong H (2018) Mining stack overflow for program repair. In: 2018 IEEE 25th international conference on software analysis evolution and reengineering, pp 118–129

Maalej W, Happel H-J (2010) Can development work describe itself?. In: 2010 7th IEEE working conference on mining software repositories. IEEE, pp 191–200

McHugh ML (2012) Interrater reliability:, The kappa statistic. Biochemia medica: Biochemia medica 22(3):276–282 MathSciNetCrossRef

Menzies T, Caglayan B, Kocaguneli E, Krall J, Peters F, Turhan B (2012) The promise repository of empirical software engineering data

Nguyen AT, Nguyen TT, Nguyen HA, Nguyen TN (2012) Multi-layered approach for recovering links between bug reports and fixes. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering, ser. FSE ’12. ACM, New York, pp 63:1–63:11. [Online]. Available: https://doi.org/10.1145/2393596.2393671

Ohira M, Kashiwa Y, Yamatani Y, Yoshiyuki H, Maeda Y, Limsettho N, Fujino K, Hata H, Ihara A, Matsumoto K (2015) A dataset of high impact bugs: Manually-classified issue reports. In: 2015 IEEE/ACM 12th working conference on mining software repositories. IEEE, pp 518–521

Online artifact page https://doi.org/10.5281/zenodo.4458586

Ponzanelli L, Bavota G, Di Penta M, Oliveto R, Lanza M (2014) Mining Stack Overflow to turn the IDE into a self-confident programming prompter. In: Proceedings of the 11th working conference on mining software repositories, ser. MSR ’14. ACM, New York, pp 102–111. [Online]. Available: https://doi.org/10.1145/2597073.2597077

Radu A, Nadi S (2019) A dataset of non-functional bugs. In: Proceedings of the 16th international conference on mining software repositories, ser. MSR ’19. IEEE Press, Piscataway, pp 399–403. [Online]. Available: https://doi.org/10.1109/MSR.2019.00066

Ragkhitwetsagul C, Krinke J, Paixao M, Bianco G, Oliveto R (2019) Toxic code snippets on stack overflow. IEEE Trans Softw Eng 47(3):560–581. https://doi.org/10.1109/TSE.2019.2900307 CrossRef

Rahman MM, Roy CK, Lo D (2019) Automatic query reformulation for code search using crowdsourced knowledge. Empir Softw Eng 24(4):1869–1924 CrossRef

Rao S, III HD (2018) Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. [Online]. Available: arXiv: 1805.04655.

Rastkar S., Murphy GC, Murray G (2010) Summarizing software artifacts:, A case study of bug reports. In: 2010 ACM/IEEE 32nd international conference on software engineering, vol 1, pp 505–514

Saifullah CMK, Asaduzzaman M, Roy CK (2019) Learning from examples to find fully qualified names of API elements in code snippets. In: 2019 34th IEEE/ACM international conference on automated software engineering, pp 243–254

Śliwerski J, Zimmermann T, Zeller A (2005) When do changes induce fixes?. In: Proceedings of the 2005 international workshop on mining software repositories, ser. MSR ’05. ACM, New York, pp 1–5. [Online]. Available: https://doi.org/10.1145/1082983.1083147

Soni A, Nadi S (2019) Analyzing comment-induced updates on stack overflow. In: Proceedings of the 16th international conference on mining software repositories, ser. MSR ’19. IEEE Press, Piscataway, pp 220–234. [Online]. Available: https://doi.org/10.1109/MSR.2019.00044

Subramanian S, Inozemtseva L, Holmes R (2014) Live API documentation. In: Proceedings of the 36th international conference on software engineering, ser. ICSE ’14. ACM, New York, pp 643–652. [Online]. Available: https://doi.org/10.1145/2568225.2568313

Subramanian S, Inozemtseva L, Holmes R (2014) Live API documentation. In: Proceedings of the 36th international conference on software engineering, pp 643–652

Swanson EB (1976) The dimensions of maintenance. In: Proceedings of the 2nd international conference on Software engineering, pp 492–497

Thiselton E, Treude C (2019) Enhancing Python compiler error messages via Stack Overflow, [Online]. Available: arXiv: 1906.11456

Tomassi DA, Dmeiri N, Wang Y, Bhowmick A, Liu Y-C, Devanbu PT, Vasilescu B, Rubio-González C (2019) Bugswarm: Mining and continuously growing a dataset of reproducible failures and fixes. In: 2019 IEEE/ACM 41st international conference on software engineering. IEEE, pp 339–349

Treude C, Robillard MP (2016) Augmenting API documentation with insights from stack overflow. In: 2016 IEEE/ACM 38th international conference on software engineering, pp 392–403

Viera AJ, Garrett JM et al (2005) Understanding interobserver agreement: the kappa statistic. Fam med 37(5):360–363

Wang S, Chen T-HP, Hassan AE (2018) How do users revise answers on technical Q&A websites? A case study on Stack Overflow. IEEE Trans Softw Eng PP:1–1

Wong AW, Salimi A, Chowdhury S, Hindle A (2019) Syntax and stack overflow: A methodology for extracting a corpus of syntax errors and fixes. In: 2019 IEEE international conference on software maintenance and evolution, pp 318–322

Zhang H, Wang S, Chen T, Hassan AE (2019a) Reading answers on Stack overflow: not enough!. IEEE Trans Softw Eng 1–1

Zhang H, Wang S, Chen TP, Zou Y, Hassan AE (2019b) An empirical study of obsolete answers on stack overflow. IEEE Trans Softw Eng 47 (4):850–862. https://doi.org/10.1109/TSE.2019.2906315

Zhang T, Yang D, Lopes C, Kirnt M (2019c) Analyzing and supporting adaptation of online code examples. In: Proceedings of the 41st international conference on software engineering, ser. ICSE ’19. IEEE Press, Piscataway, pp 316–327. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00046

Title: On using Stack Overflow comment-edit pairs to recommend code maintenance changes
Authors: Henry Tang
Sarah Nadi
Publication date: 01-07-2021
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 4/2021
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-021-09954-8

Disciplines

Events

Books

Journals

Start single access now

Request access for companies

Springer Professional

Hint

Swipe to navigate through the articles of this issue

On using Stack Overflow comment-edit pairs to recommend code maintenance changes

Abstract

To get access to this content you need the following product:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Premium Partner

Disciplines Chevron down icon Chevron up icon

Events

Books

Journals

Start single access now

Request access for companies

Springer Professional

Hint

Swipe to navigate through the articles of this issue

Abstract

Please log in to get access to this content

To get access to this content you need the following product:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 4/2021

Security assurance cases—state of the art of an emerging approach

Exposed! A case study on the vulnerability-proneness of Google Play Apps

Promises and challenges of microservices: an exploratory study

Genetic programming for feature model synthesis: a replication study

Generating API tags for tutorial fragments from Stack Overflow

Studying backers and hunters in bounty issue addressing process of open source projects

Premium Partner

Disciplines