Top

Empirical Software Engineering

Published in:

24-04-2020

What do Programmers Discuss about Deep Learning Frameworks

Authors: Junxiao Han, Emad Shihab, Zhiyuan Wan, Shuiguang Deng, Xin Xia

Published in: Empirical Software Engineering | Issue 4/2020

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Deep learning has gained tremendous traction from the developer and researcher communities. It plays an increasingly significant role in a number of application domains. Deep learning frameworks are proposed to help developers and researchers easily leverage deep learning technologies, and they attract a great number of discussions on popular platforms, i.e., Stack Overflow and GitHub. To understand and compare the insights from these two platforms, we mine the topics of interests from these two platforms. Specifically, we apply Latent Dirichlet Allocation (LDA) topic modeling techniques to derive the discussion topics related to three popular deep learning frameworks, namely, Tensorflow, PyTorch and Theano. Within each platform, we compare the topics across the three deep learning frameworks. Moreover, we make a comparison of topics between the two platforms. Our observations include 1) a wide range of topics that are discussed about the three deep learning frameworks on both platforms, and the most popular workflow stages are Model Training and Preliminary Preparation. 2) the topic distributions at the workflow level and topic category level on Tensorflow and PyTorch are always similar while the topic distribution pattern on Theano is quite different. In addition, the topic trends at the workflow level and topic category level of the three deep learning frameworks are quite different. 3) the topics at the workflow level show different trends across the two platforms. e.g., the trend of the Preliminary Preparation stage topic on Stack Overflow comes to be relatively stable after 2016, while the trend of it on GitHub shows a stronger upward trend after 2016. Besides, the Model Training stage topic still achieves the highest impact scores across two platforms. Based on the findings, we also discuss implications for practitioners and researchers.

previous article How does combinatorial testing perform in the real world: an empirical study

next article Characterizing the evolution of statically-detectable performance issues of Android apps

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

https://caffe2.ai/

https://chainer.org/

https://keras.io/

https://api.github.com

https://github.com/tensorflow/tensorflow

https://github.com/pytorch/pytorch

https://github.com/theano/theano

https://radimrehurek.com/gensim/

Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: A system for large-scale machine learning. In: OSDI, vol 16, pp 265–283

Allamanis M, Sutton C (2013) Why, when, and what: Analyzing stack overflow questions by topic, type, and code. In: Proceedings of the 10th working conference on mining software repositories, IEEE Press, pp 53–56

Azad S, Rigby P C, Guerrouj L (2017) Generating api call rules from version history and stack overflow posts. ACM Trans Softw Eng Methodol (TOSEM) 25(4):29CrossRef

Bahrampour S, Ramakrishnan N, Schott L, Shah M (2015) Comparative study of deep learning software frameworks. arXiv:151106435

Bajaj K, Pattabiraman K, Mesbah A (2014) Mining questions asked by web developers. In: Proceedings of the 11th working conference on mining software repositories, ACM, pp 112–121

Baltes S, Dumani L, Treude C, Diehl S (2018) Sotorrent: Reconstructing and analyzing the evolution of stack overflow posts. arXiv:180307311

Barua A, Thomas S W, Hassan A E (2014) What are developers talking about? an analysis of topics and trends in stack overflow. Empir Softw Eng 19(3):619–654CrossRef

Bergstra J, Breuleux O, Bastien F, Lamblin P, Pascanu R, Desjardins G, Turian J, Warde-Farley D, Bengio Y (2010) Theano: A cpu and gpu math compiler in python. In: Proc. 9th python in science conf, vol 1

Blei D M, Ng A Y, Jordan M I (2012) Latent dirichlet allocation. J Mach Learn Res 3:993–1022MATH

Both A, Hinneburg A (2015) Exploring the space of topic coherence measures. In: 8th ACM international conference on web search and data mining, pp 399–408

Bovens L, Hartmann S (2010) Bayesian Epistemology. Clarendon

Cai R, Xu B, Yang X, Zhang Z, Li Z (2017) An encoder-decoder framework translating natural language to database queries. arXiv:171106061

Chen C, Gao S, Xing Z (2016) Mining analogical libraries in q&a discussions–incorporating relational and categorical knowledge into word embedding. In: 2016 IEEE 23rd international conference on software analysis, evolution, and Reengineering (SANER), IEEE, vol 1, pp 338–348

Chen T H, Thomas S W, Hemmati H, Nagappan M, Hassan A E (2017) An empirical study on the effect of testing on code quality using topic models: A case study on software development systems. IEEE Trans Reliab R 66(3):806–824CrossRef

Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: A matlab-like environment for machine learning. In: BigLearn, NIPS workshop, EPFL-CONF-192376

De Lucia A, Di Penta M, Oliveto R, Panichella A, Panichella S (2014) Labeling source code with information retrieval methods: An empirical study. Empir Softw Eng 19(5):1383–1420CrossRef

Ding W, Wang R, Mao F, Taylor G (2014) Theano-based large-scale visual recognition with multiple gpus. arXiv:14122302

Duan C, Cui L, Chen X, Wei F, Zhu C, Zhao T (2018) Attention-fused deep matching network for natural language inference. In: IJCAI, pp 4033–4040

Erickson B J, Korfiatis P, Akkus Z, Kline T, Philbrick K (2017) Toolkits and libraries for deep learning. J Digit Imaging 30(4):400–405CrossRef

Hannun A, Case C, Casper J, Catanzaro B, Diamos G, Elsen E, Prenger R, Satheesh S, Sengupta S, Coates A et al (2014) Deep speech: Scaling up end-to-end speech recognition. arXiv:14125567

Hinton G E, Osindero S, Teh Y W (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554MathSciNetCrossRef

Hoang CDV, Haffari G, Cohn T (2017) Towards decoding as continuous optimisation in neural machine translation. In: Proceedings of the 2017 conference on empirical methods in natural language processing, pp 146–156

Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780CrossRef

Hoffman MD, Blei DM, Bach F (2010) Online learning for latent dirichlet allocation. In: International conference on neural information processing systems, pp 856–864

Ketkar N (2017) Introduction to pytorch. In: Deep learning with python, Springer, pp 195–208

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436CrossRef

Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A P, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol 2, p 4

Lee SR, Heo MJ, Lee CG, Kim M, Jeong G (2017) Applying deep learning based automatic bug triager to industrial projects. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, ACM, pp 926–931

Li H, Xing Z, Peng X, Zhao W (2013) What help do developers seek, when and how? In: 2013 20th Working Conference on Reverse Engineering (WCRE), IEEE, pp 142–151

Li H, Chen T H P, Shang W, Hassan A E (2018) Studying software logging using topic models. Empir Softw Eng 23(5):2655–2694CrossRef

Li M, Andersen D G, Park J W, Smola A J, Ahmed A, Josifovski V, Long J, Shekita E J, Su B Y (2014) Scaling distributed machine learning with the parameter server. In: OSDI, vol 14, pp 583–598

Liu H, Xu Z, Zou Y (2018) Deep learning based feature envy detection. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 385–396

Loper E, Bird S (2002) Nltk: The natural language toolkit. In: Proceedings of the ACL-02 workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics-Volume 1, Association for Computational Linguistics, pp 63–70

Lukins SK, Kraft NA, Etzkorn LH (2008) Source code retrieval for bug localization using latent dirichlet allocation. In: Working conference on reverse engineering, 2008. Wcre ’08, pp 155–164

Miller G A (1995) Wordnet: A lexical database for english. Commun ACM 38 (11):39–41CrossRef

Mo W, Shen B, Chen Y, Zhu J (2015) Tbil: A tagging-based approach to identity linkage across software communities. In: Software Engineering Conference (APSEC) 2015 Asia-Pacific, IEEE, pp 56–63

Newman D, Lau J H, Grieser K, Baldwin T (2010) Automatic evaluation of topic coherence. In: Human language technologies: Conference of the North American chapter of the association of computational linguistics, Proceedings, June 2-4, 2010 Los Angeles, California, USA, pp 100–108

Nguyen AT, Nguyen TT, Al-Kofahi J, Nguyen HV, Nguyen TN (2011) A topic-based approach for narrowing the search space of buggy files from a bug report. In: Proceedings of the 2011 26th IEEE/ACM international conference on automated software engineering, IEEE Computer Society, pp 263–272

Panichella A, Dit B, Oliveto R, Di Penta M, Poshyvanyk D, De Lucia A (2013) How to effectively use topic models for software engineering tasks? an approach based on genetic algorithms. In: Proceedings of the 2013 international conference on software engineering, IEEE Press, pp 522–531

Rosen C, Shihab E (2016) What are mobile developers asking about? a large scale study using stack overflow. Empir Softw Eng 21(3):1192–1223CrossRef

Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252MathSciNetCrossRef

Schmidhuber J (2015) Deep learning in neural networks: An overview. Neural Netw 61:85–117CrossRef

Schütze H, Manning CD, Raghavan P (2008) Introduction to information retrieval, vol 39. Cambridge University Press, CambridgeMATH

Spencer D (2009) Card sorting: Designing usable categories. Rosenfeld Media

Thomas S (2012) Mining unstructured software repositories using ir models

Treude C, Robillard MP (2016) Augmenting api documentation with insights from stack overflow. In: 2016 IEEE/ACM 38th international conference on software engineering (ICSE), IEEE, pp 392–403

Treude C, Barzilay O, Storey MA (2011) How do programmers ask and answer questions on the web?: Nier track. In: 2011 33rd international conference on software engineering (ICSE), IEEE, pp 804–807

Vasilescu B, Filkov V, Serebrenik A (2013) Stackoverflow and github: Associations between software development and crowdsourced knowledge. In: 2013 international conference on social computing (SocialCom), IEEE, pp 188–195

Wan Y, Zhao Z, Yang M, Xu G, Ying H, Wu J, Yu PS (2018) Improving automatic source code summarization via deep reinforcement learning. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 397–407

Wan Z, Lo D, Xia X, Cai L (2017) Bug characteristics in blockchain systems: A large-scale empirical study

Wan Z, Xia X, Hassan A E (2019) What do programmers discuss about blockchain? a case study on the use of balanced lda and the reference architecture of a domain to capture online discussions about blockchain platforms across stack exchange communities. IEEE Trans Softw Eng 2019:1–1

Wang S, Chen T H, Hassan A E (2018) Understanding the factors for fast answers in technical q&a websites. Empir Softw Eng 23(3):1552–1593CrossRef

Weng R, Huang S, Zheng Z, Dai X, Chen J (2017) Neural machine translation with word predictions. arXiv:170801771

Yang X L, Lo D, Xia X, Wan Z Y, Sun J L (2016) What security questions do developers ask? a large-scale study of stack overflow posts. J Comput Sci Technol 31(5):910–924CrossRef

Yao Z, Weld DS, Chen WP, Sun H (2018) Staqc: A systematically mined question-code dataset from stack overflow. arXiv:180309371

Ye D, Xing Z, Foo C Y, Li J, Kapre N (2016) Learning to extract api mentions from informal natural language discussions. In: 2016 IEEE international conference on software maintenance and evolution (ICSME), IEEE, pp 389–399

Yu L, Mishra A, Mishra D (2014) An empirical study of the dynamics of github repository and its impact on distributed software development. In: OTM confederated international conferences” on the move to meaningful internet systems”, Springer, pp 457–466

Zagalsky A, German D M, Storey M A, Teshima C G, Poo-Caamaño G (2018) How the r community creates and curates knowledge: an extended study of stack overflow and mailing lists. Empir Softw Eng 23(2):953–986CrossRef

Zhang Y, Chen Y, Cheung S C, Xiong Y, Zhang L (2018) An empirical study on tensorflow program bugs

Title: What do Programmers Discuss about Deep Learning Frameworks
Authors: Junxiao Han
Emad Shihab
Zhiyuan Wan
Shuiguang Deng
Xin Xia
Publication date: 24-04-2020
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 4/2020
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-020-09819-6

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 4/2020

Explicit programming strategies

RePOR: Mimicking humans on refactoring tasks. Are we there yet?

What am I testing and where? Comparing testing procedures based on lightweight requirements annotations

Code and commit metrics of developer productivity: a study on team leaders perceptions

An approach and benchmark to detect behavioral changes of commits in continuous integration

Characterizing the evolution of statically-detectable performance issues of Android apps

Premium Partner