Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Empirical Software Engineering
Published in: Empirical Software Engineering 2/2023

01-03-2023

An interview study about the use of logs in embedded software engineering

Authors: Nan Yang, Pieter Cuijpers, Dennis Hendriks, Ramon Schiffelers, Johan Lukkien, Alexander Serebrenik

Published in: Empirical Software Engineering | Issue 2/2023

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Context

Execution logs capture the run-time behavior of software systems. To assist developers in their maintenance tasks, many studies have proposed tools to analyze execution information from logs. However, it is as yet unknown how industry developers use logs in embedded software engineering.

Objective

In this study, we aim to understand how developers use logs in an embedded software engineering context. Specifically, we would like to gain insights into the type of logs developers analyze, the purposes for which developers analyze logs, the information developers need from logs and their expectation on tool support.

Method

In order to achieve the aim, we conducted these interview studies. First, we interviewed 25 software developers from ASML, which is a leading company in developing lithography machines. This exploratory case study provides the preliminary findings. Next, we validated and refined our findings by conducting a replication study. We involved 14 interviewees from four companies who have different software engineering roles in their daily work.

Results

As the result of our first study, we compile a preliminary taxonomy which consists of four types of logs used by developers in practice, 18 purposes of using logs, 13 types of information developers search in logs, 13 challenges faced by developers in log analysis and three suggestions for tool support provided by developers. This taxonomy is refined in the replication study with three additional purposes, one additional information need, four additional challenges and three additional suggestions of tool support. In addition, with these two studies, we observed that text-based editors and self-made scripts are commonly used when it comes to tooling in log analysis practice. As indicated by the interviewees, the development of automatic analysis tools is hindered by the quality of the logs, which further suggests several challenges in log instrumentation and management.

Conclusions

Based on our study, we provide suggestions for practitioners on logging practices. We provide implications for tool builders on how to further improve tools based on existing techniques. Finally, we suggest some research directions and studies for researchers to further study software logging.
previous article A multi-model framework for semantically enhancing detection of quality-related bug report descriptions
next article Towards understanding quality challenges of the federated learning for neural networks: a first look from the lens of robustness

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now
Footnotes
1
Company B and C are developing different kinds of consumer electronics.
 
2
cycle time is the time spent on producing an item
 
Literature
Aggarwal K, Timbers F, Rutgers T, Hindle A, Stroulia E, Greiner R (2017) Detecting duplicate bug reports with software engineering domain knowledge. J Softw Evol Process 29(3):e1821CrossRef
Amar H, Bao L, Busany N, Lo D, Maoz S (2018) Using finite-state models for log differencing. In: ESEC/FSE, pp 49–59
Antonino PO, Morgenstern A, Kuhn T (2016) Embedded-software architects: it’s not only about the software. IEEE Softw 33(6):56–62CrossRef
Artho C, Havelund K, Honiden S (2007) Visualization of concurrent program executions. In: COMPSAC, pp 541–546
Asadollah SA, Inam R, Hansson H (2015) A survey on testing for cyber physical system. In: ICTSS, pp 194–207
Asuncion H, Asuncion A, Taylor R (2010) Software traceability with topic modeling. In: ICSE (1), pp 95–104
Baltes S, Ralph P (2020) Sampling in software engineering research: a critical review and guidelines. arXiv:200207764
Bao L, Busany N, Lo D, Maoz S (2019) Statistical log differencing. In: ASE, pp 851–862
Barik T, DeLine R, Drucker S, Fisher D (2016) The bones of the system: a case study of logging and telemetry at microsoft. In: 2016 IEEE/ACM 38th international conference on software engineering companion (ICSE-C). IEEE, pp 92–101
Beschastnikh I, Liu P, Xing A, Wang P, Brun Y, Ernst MD (2020b) Visualizing distributed system executions. TOSEM 29(2):1–38CrossRef
Biermann A, Feldman J (1972) On the synthesis of finite-state machines from samples of their behavior. IEEE Trans Comput 100(6):592–597MathSciNetCrossRefMATH
Bird C (2016) Interviews. In: Perspectives on data science for software engineering. Morgan Kaufmann
Broadfoot GH (2005) Asd case notes: costs and benefits of applying formal methods to industrial control software. In: International symposium on formal methods. Springer, pp 548–551
Buchbinder E (2011) Beyond checking: experiences of the validation interview. Qual Soc Work 10(1):106–122CrossRef
Chen B, Jiang ZM (2021) A survey of software log instrumentation. ACM Comput Surv (CSUR) 54(4):1–34
Chen B, et al. (2017) Characterizing logging practices in java-based open source software projects—a replication study in apache software foundation. Empir Softw Eng 22(1):330–374CrossRef
Cicchetti A, Di Ruscio D, Eramo R, Pierantonio A (2008) Automating co-evolution in model-driven engineering. In: 2008 12th International IEEE enterprise distributed object computing conference. IEEE, pp 222–231
da Silva AJ, Linhares MV, Padilha R, Roqueiro N, de Oliveira RS (2006) An empirical study of sysml in the modeling of embedded systems. In: 2006 IEEE international conference on systems, man and cybernetics, vol 6. IEEE, pp 4569–4574
Dallmeier V, Knopp N, Mallon C, Hack S, Zeller A (2010) Generating test cases for specification mining. In: ISSTA, pp 85–96
El-Masri D, Petrillo F, Guéhéneuc YG, Hamou-Lhadj A, Bouziane A (2020) A systematic literature review on automated log abstraction techniques. Inf Softw Technol 122:106276CrossRef
Etien A, Salinesi C (2005) Managing requirements in a co-evolution context. In: 13th IEEE international 31 conference on requirements engineering (RE’05). IEEE, pp 125–134
Feng Y, Dreef K, Jones JA, van Deursen A (2018) Hierarchical abstraction of execution traces for program comprehension. In: ICPC, pp 86–96
Fidge CJ (1987) Timestamps in message-passing systems that preserve the partial ordering. Australian National University, Department of Computer Science
Fleming SD, Stirewalt R (2009) Successful strategies for debugging concurrent software: an empirical investigation. Michigan State University, Computer Science
Fluri B, Wursch M, PInzger M, Gall H (2007) Change distilling: tree differencing for fine-grained source code change extraction. TSE 33 (11):725–743
Flyvbjerg B (2007) Five misunderstandings about case-study research. Sage
Fu Q, Zhu J, Hu W, Lou JG, Ding R, Lin Q, Zhang D, Xie T (2014) Where do developers log? An empirical study on logging practices in industry. In: ICSE, pp 24–33
Gait J (1986) A probe effect in concurrent programs. Softw Pract Exp 16(3):225–233CrossRef
Gholamian S, Ward PA (2021a) A comprehensive survey of logging in software: from logging statements automation to log mining and analysis. arXiv:211012489
Gholamian S, Ward PA (2021b) What distributed systems say: a study of seven spark application logs. In: 2021 40th International symposium on reliable distributed systems (SRDS). IEEE, pp 222–232
Goldstein M, Raz D, Segall I (2017) Experience report: log-based behavioral differencing. In: ISSRE, pp 282–293
Graaf B, Lormans M, Toetenel H (2003) Embedded software engineering: the state of the practice. IEEE Softw 20(6):61–69CrossRef
Gu S, Rong G, Zhang H, Shen H (2022) Logging practices in software engineering: a systematic mapping study. IEEE Trans Softw Eng
Gulzar MA, Zhu Y, Han X (2019) Perception and practices of differential testing. In: 2019 IEEE/ACM 41st international conference on software engineering: software engineering in practice (ICSE-SEIP). IEEE, pp 71–80
Gupta M, Mandal A, Dasgupta G, Serebrenik A (2018) Runtime monitoring in continuous deployment by differencing execution behavior model. In: Pahl C, Vukovic M, Yin J, Yu Q (eds) ICSOC, Springer, Lecture Notes in Computer Science. https://​doi.​org/​10.​1007/​978-3-030-03596-9_​58, vol 11236, pp 812–827
Harty J, Zhang H, Wei L, Pascarella L, Aniche M, Shang W (2021) Logging practices with mobile analytics: An empirical study on firebase. In: 2021 IEEE/ACM 8th international conference on mobile software engineering and systems (MobileSoft). IEEE, pp 56–60
He S, He P, Chen Z, Yang T, Su Y, Lyu MR (2021) A survey on automated log analysis for reliability engineering. ACM Comput Surv (CSUR) 54(6):1–37CrossRef
Hendriks D, Meer Avd, Oortwijn W (2022) A multi-level methodology for behavioral comparison of software-intensive systems. In: International conference on formal methods for industrial critical systems. Springer, pp 226–243
Henzinger TA, Sifakis J (2007) The discipline of embedded systems design. Computer 40(10):32–40CrossRef
Holton JA (2007) The coding process and its challenges. The Sage handbook of grounded theory, vol 3, pp 265–289
Hooimeijer B, Geilen M, Groote JF, Hendriks D, Schiffelers R (2022) Constructive model inference: model learning for component-based software architectures. In: 17th International conference on software technologies (ICSOFT), pp 146–158
Jerding DF, Stasko JT, Ball T (1997) Visualizing interactions in program executions. In: ICSE, pp 360–370
Kabinna S, Bezemer CP, Shang W, Hassan AE (2016) Logging library migrations: a case study for the apache software foundation projects. In: 2016 IEEE/ACM 13th working conference on mining software repositories (MSR). IEEE, pp 154–164
Kabinna S, Bezemer CP, Shang W, Syer MD, Hassan AE (2018) Examining the stability of logging statements. Empir Softw Eng 23(1):290–333CrossRef
Kagdi H, Gethers M, Poshyvanyk D, Collard ML (2010) Blending conceptual and evolutionary couplings to support change impact analysis in source code. In: RE, pp 119–128
Krka I, Brun Y, Medvidovic N (2014) Automatic mining of specifications from invocation traces and method invariants. In: ESEC/FSE, pp 178–189
Kurfess TR, Hodgson TJ (2007) Metrology, sensors and control. In: Micromanufacturing. Springer, pp 89–109
Lee EA (2008) Cyber physical systems: design challenges. In: 2008 11th IEEE international symposium on object and component-oriented real-time distributed computing (ISORC). IEEE, pp 363–369
Legunsen O, Hassan WU, Xu X, Roşu G, Marinov D (2016) How good are the specs? A study of the bug-finding effectiveness of existing java api specifications. In: ASE, pp 602–613
Lemieux C, Park D, Beschastnikh I (2015) General ITL specification mining. In: ASE, pp 81–92
Levy O, Feitelson D (2019) Understanding large-scale software—a hierarchical view. In: ICPC, pp 283–293
Li H, Chen THP, Shang W, Hassan AE (2018) Studying software logging using topic models. EMSE 23(5):2655–2694
Li H, Shang W, Adams B, Sayagh M, Hassan AE (2020a) A qualitative study of the benefits and costs of logging from developers’ perspectives. TSE
Li S, Niu X, Jia Z, Liao X, Wang J, Li T (2020b) Guiding log revisions by learning from software evolution history. Empir Softw Eng 25 (3):2302–2340CrossRef
Lian X, Rahimi M, Cleland-Huang J, Zhang L, Ferrai R, Smith M (2016) Mining requirements knowledge from collections of domain documents. In: 2016 IEEE 24th international requirements engineering conference (RE), pp 156–165
Liu X, Lin W, Pan A, Zhang Z (2007) WiDS checker: combating bugs in distributed systems. In: 4th Symposium on networked systems design and implementation, NSDI 2007, pp 257–270
Lo D, Maoz S (2012) Scenario-based and value-based specification mining: better together. In: ASE, vol 19, pp 423–458
Lo D, Mariani L, Pezzè M (2009) Automatic steering of behavioral model inference. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, pp 345–354
Luo Q, Hariri F, Eloussi L, Marinov D (2014) An empirical analysis of flaky tests. In: FSE, pp 643–653
Maoz S, Ringert JO, Rumpe B (2010) A manifesto for semantic model differencing. In: MODELS. Springer, pp 194–203
Mashhadi MJ, Hemmati H (2019) An empirical study on practicality of specification mining algorithms on a real-world application. In: ICPC, pp 65–69
Mashhadi MJ, Siddiqui TR, Hemmati H, Loewen H (2019) Interactive semi-automated specification mining for debugging: an experience report. Inf Softw Technol 113:20–38CrossRef
Mattern F, et al. (1988) Virtual time and global states of distributed systems. Univ. Department of Computer Science
McBurney PW, Liu C, McMillan C, Weninger T (2014) Improving topic model source code summarization. In: Proceedings of the 22nd international conference on program comprehension, pp 291–294
Mengerink J, Schiffelers RRH, Serebrenik A, van den Brand M (2016) Dsl/model co-evolution in industrial emf-based MDSE ecosystems. In: Mayerhofer T, Pierantonio A, Schätz B, Tamzalit D (eds) Proceedings of the 10th workshop on models and evolution co-located with ACM/IEEE 19th international conference on model driven engineering languages and systems (MODELS 2016), Saint-Malo, France, October 2, 2016, CEUR-WS.org, CEUR Workshop Proceedings, vol 1706. http://​ceur-ws.​org/​Vol-1706/​paper1.​pdf, pp 2–7
Mishra D, Mishra A, Yazici A (2008) Successful requirement elicitation by combining requirement engineering techniques. In: 2008 First international conference on the applications of digital information and Web technologies (ICADIWT). IEEE, pp 258–263
Noergaard T (2012) Embedded systems architecture: a comprehensive guide for engineers and programmers. Newnes
Pandey D, Suman U, Ramani AK (2010) An effective requirement engineering process model for software development and requirements management. In: 2010 International conference on advances in recent technologies in communication and computing. IEEE, pp 287–291
Pecchia A, Cinque M, Carrozza G, Cotroneo D (2015) Industry practices and event logging: assessment of a critical software development process. In: ICSE (2). IEEE, pp 169–178
Pradel M, Gross TR (2012) Leveraging test generation and specification mining for automated bug detection without false positives. In: ICSE, pp 288–298
Rong G, Zhang Q, Liu X, Gu S (2017) A systematic review of logging practice in software engineering. In: 2017 24th Asia-Pacific software engineering conference (APSEC). IEEE, pp 534–539
Rong G, Gu S, Zhang H, Shao D, Liu W (2018) How is logging practice implemented in open source software projects? A preliminary exploration. In: 2018 25th Australasian software engineering conference (ASWEC). IEEE, pp 171–180
Rong G, Xu Y, Gu S, Zhang H, Shao D (2020) Can you capture information as you intend to? A case study on logging practice in industry. In: 2020 IEEE International conference on software maintenance and evolution (ICSME). IEEE, pp 12–22
Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131CrossRef
Said W, Quante J, Koschke R (2018) Towards interactive mining of understandable state machine models from embedded software. In: MODELSWARD, pp 117–128
Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng 25(4):557–572CrossRef
Shaft TM, Vessey I (1995) The relevance of application domain knowledge: the case of computer program comprehension. ISR 6(3):286–299CrossRef
Shang W, Nagappan M, Hassan AE, Jiang ZM (2014) Understanding log lines using development knowledge. In: ICSME, pp 21–30
Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218CrossRef
Silva E, Freitas EP, Wagner FR, Carvalho FC, Pereira CE (2006) Java framework for distributed real-time embedded systems. In: 9th IEEE international symposium on object and component-oriented real-time distributed computing (ISORC’06). IEEE, p 8
Storey MA, Ernst NA, Williams C, Kalliamvakou E (2020) The who, what, how of software engineering research: a socio-technical framework. Empir Softw Eng 25(5):4097–4129CrossRef
Strandberg PE, Enoiu EP, Afzal W, Sundmark D, Feldt R (2019) Information flow in software testing—an interview study with embedded software engineering practitioners. IEEE Access 7:46434–46453CrossRef
Strauss A, Corbin JM (1997) Grounded theory in practice. Sage
Tan L, Bockisch C (2019) A survey of refactoring detection tools. In: EMLS, pp 100–105
van der Werf JMEM, van Dongen BF, Hurkens CAJ, Serebrenik A (2009) Process Discovery using Integer Linear Programming. Fundam Inform 94 (3-4):387–412MathSciNetCrossRefMATH
Vogel-Heuser B, Fay A, Schaefer I, Tichy M (2015) Evolution of software in automated production systems: challenges and research directions. J Syst Softw 110:54–84CrossRef
Walkinshaw N, Taylor R, Derrick J (2016) Inferring extended finite state machine models from software executions. Empir Softw Eng 21(3):811–853CrossRef
Yang N, Aslam K, Schiffelers R, Lensink L, Hendriks D, Cleophas L, Serebrenik A (2019) Improving model inference in industry by combining active and passive learning. In: SANER, pp 253–263
Yang N, Cuijpers PJL, Schiffelers RRH, Lukkien J, Serebrenik A (2021) An interview study of how developers use execution logs in embedded software engineering. In: 43rd IEEE/ACM International conference on software engineering: software engineering in practice, ICSE (SEIP) 2021, Madrid, Spain, May 25–28, 2021. https://​doi.​org/​10.​1109/​ICSE-SEIP52600.​2021.​00015. IEEE, pp 61–70
Ye X, Bunescu R, Liu C (2014) Learning to rank relevant files for bug reports using domain knowledge. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp 689–699
Yuan D, Park S, Zhou Y (2012) Characterizing logging practices in open-source software. In: 2012 34th International conference on software engineering (ICSE). IEEE, pp 102–112
Zaidman A, Demeyer S (2004) Managing trace data volume through a heuristical clustering process based on event execution frequency. In: CSMR, pp 329–338
Zaidman A, Van Rompaey B, van Deursen A, Demeyer S (2011) Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empir Softw Eng 16(3):325–364CrossRef
Zeng Y, Chen J, Shang W, Chen THP (2019) Studying the characteristics of logging practices in mobile apps: a case study on f-droid. Empir Softw Eng 24(6):3394–3434CrossRef
Zhu J, He P, Fu Q, Zhang H, Lyu MR, Zhang D (2015) Learning to log: helping developers make informed logging decisions. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 1. IEEE, pp 415–425
Metadata
Title
An interview study about the use of logs in embedded software engineering
Authors
Nan Yang
Pieter Cuijpers
Dennis Hendriks
Ramon Schiffelers
Johan Lukkien
Alexander Serebrenik
Publication date
01-03-2023
Publisher
Springer US
Published in
Empirical Software Engineering / Issue 2/2023
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI
https://doi.org/10.1007/s10664-022-10258-8

Other articles of this Issue 2/2023

Empirical Software Engineering 2/2023 Go to the issue

OriginalPaper

Evaluating ensemble imputation in software effort estimation

OriginalPaper

How programmers find online learning resources

OriginalPaper

Semantically-enhanced topic recommendation systems for software projects

OriginalPaper

Satisfaction and performance of software developers during enforced work from home in the COVID-19 pandemic

OriginalPaper

Evaluating state-of-the-art # SAT solvers on industrial configuration spaces

OriginalPaper

Vulnerability management in Linux distributions

Premium Partner

    Image Credits