ABSTRACT
Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of trouble-shooting any complex software system, but further exacerbated by the paucity of information that is typically available in the production setting. Indeed, for reasons of both overhead and privacy, it is common that only the run-time log generated by a system (e.g., syslog) can be shared with the developers. Unfortunately, the ad-hoc nature of such reports are frequently insufficient for detailed failure diagnosis. This paper seeks to improve this situation within the rubric of existing practice. We describe a tool, LogEnhancer that automatically "enhances" existing logging code to aid in future post-failure debugging. We evaluate LogEnhancer on eight large, real-world applications and demonstrate that it can dramatically reduce the set of potential root failure causes that must be considered during diagnosis while imposing negligible overheads.
- Cisco system log management.Google Scholar
- EMC seen collecting and managing log as key driver for 94 percent of customers.Google Scholar
- M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP'03. Google ScholarDigital Library
- A. V. Aho, M. S. Lam, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools (2nd Edition), Page 528. Google ScholarDigital Library
- A. Aiken, S. Bugrara, I. Dillig, T. Dillig, P. Hawkins, and B. Hackett. An overview of the Saturn project. In PASTE'07. Google ScholarDigital Library
- Apple Inc., CrashReport. Technical Report TN2123, 2004.Google Scholar
- A. Ayers, R. Schooler, C. Metcalf, A. Agarwal, J. Rhee, and E. Witchel. Traceback: First fault diagnosis by reconstruction of distributed control flow. In PLDI'05. Google ScholarDigital Library
- P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using magpie for request extraction and workload modelling. In OSDI'04. Google ScholarDigital Library
- E. D. Berger and B. G. Zorn. Diehard: probabilistic memory safety for unsafe languages. In PLDI'06. Google ScholarDigital Library
- S. Bhatia, A. Kumar, M. Fiuczynski, and L. Peterson. Lightweight, high-resolution monitoring for troubleshooting production systems. In OSDI'08. Google ScholarDigital Library
- C. Cadar, D. Dunbar, and D. R. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI'08. Google ScholarDigital Library
- M. Castro, M. Costa, and J.-P. Martin. Better bug reporting with better privacy. In ASPLOS'08. Google ScholarDigital Library
- S. Chen, M. Kozuch, T. Strigkos, B. Falsafi, P. B. Gibbons, T. C. Mowry, V. Ramachandran, O. Ruwase, M. Ryan, and E. Vlachos. Flexible hardware acceleration for instruction-grain program monitoring. In ISCA'08. Google ScholarDigital Library
- L. Chew and D. Lie. Kivati: fast detection and prevention of atomicity violations. In EuroSys'10. Google ScholarDigital Library
- T. M. Chilimbi, B. Liblit, K. Mehra, A. V. Nori, and K. Vaswani. HOLMES: Effective statistical debugging via efficient path profiling. In ICSE'09. Google ScholarDigital Library
- I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox. Capturing, indexing, clustering, and retrieving system history. In SOSP'05. Google ScholarDigital Library
- M. Costa, M. Castro, L. Zhou, L. Zhang, and M. Peinado. Bouncer: securing software by blocking bad input. In SOSP'07. Google ScholarDigital Library
- Dell. Streamlined Troubleshooting with the Dell system E-Support tool. Dell Power Solutions, 2008.Google Scholar
- D. L. Detlefs, K. R. M. Leino, K. Rustan, M. Leino, G. Nelson, and J. B. Saxe. Extended static checking. In TR SRC-159, COMPAQ SRC, 1998. Google ScholarDigital Library
- J. Devietti, B. Lucia, M. Oskin, and L. Ceze. Dmp: Deterministic shared-memory multiprocessing. In ASPLOS'09. Google ScholarDigital Library
- G. Dunlap, D. Lucchetti, M. Fetterman, and P. Chen. Execution replay of multiprocessor virtual machines. In VEE, 2008. Google ScholarDigital Library
- The DWARF Debugging Format. http://dwarfstd.org.Google Scholar
- D. Engler and K. Ashcraft. Racerx: effective, static detection of race conditions and deadlocks. In SOSP'03. Google ScholarDigital Library
- C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe, and R. Stata. Extended static checking for java. In PLDI'02. Google ScholarDigital Library
- Man page for gcore (Linux section 1).Google Scholar
- K. Glerum, K. Kinshumann, S. Greenberg, G. Aul, V. Orgovan, G. Nichols, D. Grant, G. Loihle, and G. Hunt. Debugging in the (very) large: ten years of implementation and experience. In SOSP'09. Google ScholarDigital Library
- Google Inc., Breakpad.Google Scholar
- Z. Guo, X. Wang, J. Tang, X. Liu, Z. Xu, M. Wu, M. F. Kaashoek, and Z. Zhang. R2: An application-level kernel for record and replay. In OSDI'08. Google ScholarDigital Library
- J. Ha, C. J. Rossbach, J. V. Davis, I. Roy, H. E. Ramadan, D. E. Porter, D. L. Chen, and E. Witchel. Improved error reporting for software that uses black-box components. In PLDI'07. Google ScholarDigital Library
- B. Hackett and A. Aiken. How is aliasing used in systems software? In FSE'06. Google ScholarDigital Library
- A. Kadav, M. J. Renzelmann, and M. M. Swift. Tolerating hardware device failures in software. In SOSP'09. Google ScholarDigital Library
- B. W. Kernighan and R. Pike. The Practice of Programming. Addison-Wesley, 1999. Google ScholarDigital Library
- S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In USENIX ATC'05. Google ScholarDigital Library
- T. J. LeBlanc and J. M. Mellor-Crummey. Debugging parallel programs with instant replay. IEEE Trans. Comput., 36(4), 1987. Google ScholarDigital Library
- D. Lee, B. Wester, K. Veeraraghavan, S. Narayanasamy, P. M. Chen, and J. Flinn. Respec: efficient online multiprocessor replay via speculation and external determinism. In ASPLOS'10. Google ScholarDigital Library
- Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Have things changed now? An empirical study of bug characteristics in modern open source software. In ASID '06: Proceedings of the 1st workshop on Architectural and system support for improving software dependability, October 2006. Google ScholarDigital Library
- B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug isolation via remote program sampling. In PLDI'03. Google ScholarDigital Library
- B. Lucia, L. Ceze, and K. Strauss. Colorsafe: architectural support for debugging and dynamically avoiding multi-variable atomicity violations. In ISCA'10. Google ScholarDigital Library
- R. Manevich, M. Sridharan, S. Adams, M. Das, and Z. Yang. PSE: Explaining program failures via postmortem static analysis. SIGSOFT Softw. Eng. Notes, 29(6):63--72, 2004. Google ScholarDigital Library
- P. Montesinos, L. Ceze, and J. Torrellas. Delorean: Recording and deterministically replaying shared-memory multiprocessor execution efficiently. In ISCA'08. Google ScholarDigital Library
- Mozilla Quality Feedback Agent. http://support.mozilla.com/en-US/kb/quality feedback agent.Google Scholar
- M. Naik and A. Aiken. Conditional must not aliasing for static race detection. In POPL'07. Google ScholarDigital Library
- M. Naik, A. Aiken, and J. Whaley. Effective static race detection for java. In PLDI'06. Google ScholarDigital Library
- S. Narayanasamy, G. Pokam, and B. Calder. Bugnet: Continuously recording program execution for deterministic replay debugging. In ISCA'05. Google ScholarDigital Library
- G. C. Necula, S. McPeak, S. P. Rahul, and W. Weimer. Cil: Intermediate language and tools for analysis and transformation of c programs. In CC'02. Google ScholarDigital Library
- NetApp Inc., Savecore. ONTAP 7.3 Manual Page Reference, Volume 1, Pages 471--472.Google Scholar
- NetApp. Proactive health management with auto-support. NetApp White Paper, 2007.Google Scholar
- G. Novark, E. D. Berger, and B. G. Zorn. Exterminator: automatically correcting memory errors with high probability. In PLDI'07. Google ScholarDigital Library
- M. Olszewski, J. Ansel, and S. Amarasinghe. Kendo: Efficient Deterministic Multithreading in software. In ASPLOS'09. Google ScholarDigital Library
- S. Park, W. Xiong, Z. Yin, R. Kaushik, K. H.Lee, S. Lu, and Y. Zhou. Pres:probabilistic replay with execution sketching on multiprocessors. In SOSP, 2009. Google ScholarDigital Library
- S. Schmidt. 7 more good tips on logging. http://codemonkeyism.com/7-more-good-tips-on-logging/.Google Scholar
- E. Vlachos, M. L. Goodstein, M. A. Kozuch, S. Chen, B. Falsafi, P. B. Gibbons, and T. C. Mowry. Paralog: enabling and accelerating online parallel monitoring of multithreaded applications. In ASPLOS'10. Google ScholarDigital Library
- VMware. Using the integrated virtual debugger for visual studio. http://www.vmware.com/pdf/ws65_manual.pdf.Google Scholar
- D. Weeratunge, X. Zhang, and S. Jagannathan. Analyzing multicore dumps to facilitate concurrency bug reproduction. SIGARCH Comput. Archit. News, 38(1):155--166, 2010. Google ScholarDigital Library
- M. Xu, R. Bodik, and M. Hill. A "flight data recorder" for enabling full-system multiprocessor deterministic replay. In ISCA'03. Google ScholarDigital Library
- W. Xu, L. Huang, M. Jordan, D. Patterson, and A. Fox. Mining console logs for large-scale system problem detection. In SOSP'09.Google Scholar
- D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy. SherLog: Error diagnosis by connecting clues from run-time logs. In ASPLOS'10. Google ScholarDigital Library
- C. Zamfir and G. Candea. Execution synthesis: a technique for automated software debugging. In EuroSys'10. Google ScholarDigital Library
- C. Zamfir and G. Candea. Low-overhead bug fingerprinting for fast debugging. In Runtime Verification, volume 6418 of Lecture Notes in Computer Science, pages 460--468. 2010. Google ScholarDigital Library
- Q. Zhao, R. Rabbah, S. Amarasinghe, L. Rudolph, and W.-F. Wong. How to do a million watchpoints: efficient debugging using dynamic instrumentation. In CC'08. Google ScholarDigital Library
Index Terms
- Improving software diagnosability via log enhancement
Recommendations
A Survey on Automated Log Analysis for Reliability Engineering
Invited TutorialLogs are semi-structured text generated by logging statements in software source code. In recent decades, software logs have become imperative in the reliability assurance mechanism of many software systems, because they are often the only data ...
Improving Software Diagnosability via Log Enhancement
Special Issue APLOS 2011Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of troubleshooting any complex software system, but further exacerbated by the paucity of information that is typically available in the ...
Improving software diagnosability via log enhancement
ASPLOS '11Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of trouble-shooting any complex software system, but further exacerbated by the paucity of information that is typically available in the ...
Comments