skip to main content
10.1145/1950365.1950369acmconferencesArticle/Chapter ViewAbstractPublication PagesasplosConference Proceedingsconference-collections
research-article

Improving software diagnosability via log enhancement

Authors Info & Claims
Published:05 March 2011Publication History

ABSTRACT

Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of trouble-shooting any complex software system, but further exacerbated by the paucity of information that is typically available in the production setting. Indeed, for reasons of both overhead and privacy, it is common that only the run-time log generated by a system (e.g., syslog) can be shared with the developers. Unfortunately, the ad-hoc nature of such reports are frequently insufficient for detailed failure diagnosis. This paper seeks to improve this situation within the rubric of existing practice. We describe a tool, LogEnhancer that automatically "enhances" existing logging code to aid in future post-failure debugging. We evaluate LogEnhancer on eight large, real-world applications and demonstrate that it can dramatically reduce the set of potential root failure causes that must be considered during diagnosis while imposing negligible overheads.

References

  1. Cisco system log management.Google ScholarGoogle Scholar
  2. EMC seen collecting and managing log as key driver for 94 percent of customers.Google ScholarGoogle Scholar
  3. M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP'03. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. A. V. Aho, M. S. Lam, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools (2nd Edition), Page 528. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. A. Aiken, S. Bugrara, I. Dillig, T. Dillig, P. Hawkins, and B. Hackett. An overview of the Saturn project. In PASTE'07. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Apple Inc., CrashReport. Technical Report TN2123, 2004.Google ScholarGoogle Scholar
  7. A. Ayers, R. Schooler, C. Metcalf, A. Agarwal, J. Rhee, and E. Witchel. Traceback: First fault diagnosis by reconstruction of distributed control flow. In PLDI'05. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using magpie for request extraction and workload modelling. In OSDI'04. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. E. D. Berger and B. G. Zorn. Diehard: probabilistic memory safety for unsafe languages. In PLDI'06. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. S. Bhatia, A. Kumar, M. Fiuczynski, and L. Peterson. Lightweight, high-resolution monitoring for troubleshooting production systems. In OSDI'08. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. C. Cadar, D. Dunbar, and D. R. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI'08. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. M. Castro, M. Costa, and J.-P. Martin. Better bug reporting with better privacy. In ASPLOS'08. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. S. Chen, M. Kozuch, T. Strigkos, B. Falsafi, P. B. Gibbons, T. C. Mowry, V. Ramachandran, O. Ruwase, M. Ryan, and E. Vlachos. Flexible hardware acceleration for instruction-grain program monitoring. In ISCA'08. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. L. Chew and D. Lie. Kivati: fast detection and prevention of atomicity violations. In EuroSys'10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. T. M. Chilimbi, B. Liblit, K. Mehra, A. V. Nori, and K. Vaswani. HOLMES: Effective statistical debugging via efficient path profiling. In ICSE'09. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox. Capturing, indexing, clustering, and retrieving system history. In SOSP'05. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. M. Costa, M. Castro, L. Zhou, L. Zhang, and M. Peinado. Bouncer: securing software by blocking bad input. In SOSP'07. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Dell. Streamlined Troubleshooting with the Dell system E-Support tool. Dell Power Solutions, 2008.Google ScholarGoogle Scholar
  19. D. L. Detlefs, K. R. M. Leino, K. Rustan, M. Leino, G. Nelson, and J. B. Saxe. Extended static checking. In TR SRC-159, COMPAQ SRC, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. J. Devietti, B. Lucia, M. Oskin, and L. Ceze. Dmp: Deterministic shared-memory multiprocessing. In ASPLOS'09. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. G. Dunlap, D. Lucchetti, M. Fetterman, and P. Chen. Execution replay of multiprocessor virtual machines. In VEE, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. The DWARF Debugging Format. http://dwarfstd.org.Google ScholarGoogle Scholar
  23. D. Engler and K. Ashcraft. Racerx: effective, static detection of race conditions and deadlocks. In SOSP'03. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe, and R. Stata. Extended static checking for java. In PLDI'02. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Man page for gcore (Linux section 1).Google ScholarGoogle Scholar
  26. K. Glerum, K. Kinshumann, S. Greenberg, G. Aul, V. Orgovan, G. Nichols, D. Grant, G. Loihle, and G. Hunt. Debugging in the (very) large: ten years of implementation and experience. In SOSP'09. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Google Inc., Breakpad.Google ScholarGoogle Scholar
  28. Z. Guo, X. Wang, J. Tang, X. Liu, Z. Xu, M. Wu, M. F. Kaashoek, and Z. Zhang. R2: An application-level kernel for record and replay. In OSDI'08. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. J. Ha, C. J. Rossbach, J. V. Davis, I. Roy, H. E. Ramadan, D. E. Porter, D. L. Chen, and E. Witchel. Improved error reporting for software that uses black-box components. In PLDI'07. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. B. Hackett and A. Aiken. How is aliasing used in systems software? In FSE'06. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. A. Kadav, M. J. Renzelmann, and M. M. Swift. Tolerating hardware device failures in software. In SOSP'09. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. B. W. Kernighan and R. Pike. The Practice of Programming. Addison-Wesley, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In USENIX ATC'05. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. T. J. LeBlanc and J. M. Mellor-Crummey. Debugging parallel programs with instant replay. IEEE Trans. Comput., 36(4), 1987. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. D. Lee, B. Wester, K. Veeraraghavan, S. Narayanasamy, P. M. Chen, and J. Flinn. Respec: efficient online multiprocessor replay via speculation and external determinism. In ASPLOS'10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Have things changed now? An empirical study of bug characteristics in modern open source software. In ASID '06: Proceedings of the 1st workshop on Architectural and system support for improving software dependability, October 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug isolation via remote program sampling. In PLDI'03. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. B. Lucia, L. Ceze, and K. Strauss. Colorsafe: architectural support for debugging and dynamically avoiding multi-variable atomicity violations. In ISCA'10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. R. Manevich, M. Sridharan, S. Adams, M. Das, and Z. Yang. PSE: Explaining program failures via postmortem static analysis. SIGSOFT Softw. Eng. Notes, 29(6):63--72, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. P. Montesinos, L. Ceze, and J. Torrellas. Delorean: Recording and deterministically replaying shared-memory multiprocessor execution efficiently. In ISCA'08. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Mozilla Quality Feedback Agent. http://support.mozilla.com/en-US/kb/quality feedback agent.Google ScholarGoogle Scholar
  42. M. Naik and A. Aiken. Conditional must not aliasing for static race detection. In POPL'07. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. M. Naik, A. Aiken, and J. Whaley. Effective static race detection for java. In PLDI'06. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. S. Narayanasamy, G. Pokam, and B. Calder. Bugnet: Continuously recording program execution for deterministic replay debugging. In ISCA'05. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. G. C. Necula, S. McPeak, S. P. Rahul, and W. Weimer. Cil: Intermediate language and tools for analysis and transformation of c programs. In CC'02. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. NetApp Inc., Savecore. ONTAP 7.3 Manual Page Reference, Volume 1, Pages 471--472.Google ScholarGoogle Scholar
  47. NetApp. Proactive health management with auto-support. NetApp White Paper, 2007.Google ScholarGoogle Scholar
  48. G. Novark, E. D. Berger, and B. G. Zorn. Exterminator: automatically correcting memory errors with high probability. In PLDI'07. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. M. Olszewski, J. Ansel, and S. Amarasinghe. Kendo: Efficient Deterministic Multithreading in software. In ASPLOS'09. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. S. Park, W. Xiong, Z. Yin, R. Kaushik, K. H.Lee, S. Lu, and Y. Zhou. Pres:probabilistic replay with execution sketching on multiprocessors. In SOSP, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. S. Schmidt. 7 more good tips on logging. http://codemonkeyism.com/7-more-good-tips-on-logging/.Google ScholarGoogle Scholar
  52. E. Vlachos, M. L. Goodstein, M. A. Kozuch, S. Chen, B. Falsafi, P. B. Gibbons, and T. C. Mowry. Paralog: enabling and accelerating online parallel monitoring of multithreaded applications. In ASPLOS'10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. VMware. Using the integrated virtual debugger for visual studio. http://www.vmware.com/pdf/ws65_manual.pdf.Google ScholarGoogle Scholar
  54. D. Weeratunge, X. Zhang, and S. Jagannathan. Analyzing multicore dumps to facilitate concurrency bug reproduction. SIGARCH Comput. Archit. News, 38(1):155--166, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. M. Xu, R. Bodik, and M. Hill. A "flight data recorder" for enabling full-system multiprocessor deterministic replay. In ISCA'03. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. W. Xu, L. Huang, M. Jordan, D. Patterson, and A. Fox. Mining console logs for large-scale system problem detection. In SOSP'09.Google ScholarGoogle Scholar
  57. D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy. SherLog: Error diagnosis by connecting clues from run-time logs. In ASPLOS'10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. C. Zamfir and G. Candea. Execution synthesis: a technique for automated software debugging. In EuroSys'10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. C. Zamfir and G. Candea. Low-overhead bug fingerprinting for fast debugging. In Runtime Verification, volume 6418 of Lecture Notes in Computer Science, pages 460--468. 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Q. Zhao, R. Rabbah, S. Amarasinghe, L. Rudolph, and W.-F. Wong. How to do a million watchpoints: efficient debugging using dynamic instrumentation. In CC'08. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Improving software diagnosability via log enhancement

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      ASPLOS XVI: Proceedings of the sixteenth international conference on Architectural support for programming languages and operating systems
      March 2011
      432 pages
      ISBN:9781450302661
      DOI:10.1145/1950365
      • cover image ACM SIGPLAN Notices
        ACM SIGPLAN Notices  Volume 46, Issue 3
        ASPLOS '11
        March 2011
        407 pages
        ISSN:0362-1340
        EISSN:1558-1160
        DOI:10.1145/1961296
        Issue’s Table of Contents
      • cover image ACM SIGARCH Computer Architecture News
        ACM SIGARCH Computer Architecture News  Volume 39, Issue 1
        ASPLOS '11
        March 2011
        407 pages
        ISSN:0163-5964
        DOI:10.1145/1961295
        Issue’s Table of Contents

      Copyright © 2011 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 5 March 2011

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      Overall Acceptance Rate535of2,713submissions,20%

      Upcoming Conference

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader