Kazuki Otomo
ABSTRACT
System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 15 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method detects anomalies and ease troubleshooting of network system faults.
- E. Baseman, S. Blanchard, and E. Zongzelimyuntedu. Relational Synthesis of Text and Numeric Data for Anomaly Detection on Computing System Logs. In Proc. IEEE ICMLA'16, pages 2--5, 2016.Google Scholar
Cross Ref
- M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A Density-based Algorithm for Discovering Clusters a Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proc. ACM KDD'96, pages 226--231, 1996. Google ScholarDigital Library
- T. Hacker, R. Pais, and C. Rong. A markov random field based approach for analyzing supercomputer system logs. IEEE Transactions on Cloud Computing, pages 1--1, 2017.Google ScholarCross Ref
- T. Kimura, K. Ishibashi, T. Mori, H. Sawada, T. Toyono, K. Nishimatsu, A. Watanabe, A. Shimoda, and K. Shiomoto. Spatio-temporal factorization of log data for understanding network events. In Proc IEEE INFOCOM'14, pages 610--618, 2014.Google ScholarCross Ref
- T. Kimura, A. Watanabe, T. Toyono, and K. Ishibashi. Proactive failure detection learning generation patterns of large-scale network logs. In Proc CNSM'15, pages 8--14, 2015. Google ScholarDigital Library
- D. P. Kingma, D. J. Rezende, S. Mohamed, and M. Welling. Semi-Supervised Learning with Deep Generative Models. pages 1--9, 2014.Google Scholar
- D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. (Ml):1--14, 2013.Google Scholar
- J. Kleinberg. Bursty and Hierarchical Structure in Streams. In Proc. ACM KDD'02, pages 91--101, 2002. Google ScholarDigital Library
- S. Kobayashi, K. Otomo, K. Fukuda, and H. Esaki. Mining causality of network events in log data. IEEE TNSM, 15(1):53--67, 2018.Google Scholar
- A. Lakhina, M. Crovella, and C. Diot. Diagnosing network-wide traffic anomalies. In Proc. ACM SIGCOMM'04, 34(4):219, 2004. Google ScholarDigital Library
- S. Lu, B. Rao, X. Wei, B. Tak, L. Wang, and L. Wang. Log-based Abnormal Task Detection and Root Cause Analysis for Spark. In Proc. IEEE ICWS'17, 2017.Google ScholarCross Ref
- M. Moh, S. Pininti, S. Doddapaneni, and T.-S. Moh. Detecting Web Attacks Using Multi-stage Log Analysis. In Proc. IEEE IACC'16, pages 733--738, 2016.Google Scholar
- K. Otomo, S. Kobayashi, K. Fukuda, and H. Esaki. An Analysis of Burstiness and Causality of System Logs. In Proc. AINTEC'17, pages 16--23, 2017. Google ScholarDigital Library
- M. Shatnawi and M. Hefeeda. Real-time failure prediction in online services. In Proc. IEEE INFOCOM'15, pages 1391--1399, 2015.Google ScholarCross Ref
- S. Urushidani, M. Aoki, K. Fukuda, S. Abe, M. Nakamura, M. Koibuchi, Y. Ji, and S. Yamada. Highly available network design and resource management of SINET4. Telecommunication Systems, 56(1):33--47, 2014. Google ScholarDigital Library
- W. Xu, L. Huang, A. Fox, D. Patterson, M. I. Jordan, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting Large-Scale System Problems by Mining Console Logs. In Proc. ACM SOSP'09, pages 117--131, 2009. Google ScholarDigital Library
- Z. Zheng, Z. Lan, B. H. Park, and A. Geist. System log pre-processing to improve failure prediction. In Prco IEEE DSN'09, pages 572--577, 2009.Google ScholarCross Ref
- J. Zhong, W. Guo, and Z. Wang. Study on network failure prediction based on alarm logs. In Proc. ICBDSC'16, pages 23--29, 2016.Google ScholarCross Ref
Index Terms
- Finding Anomalies in Network System Logs with Latent Variables
Recommendations
An Analysis of Burstiness and Causality of System Logs
AINTEC '17: Proceedings of the 13th Asian Internet Engineering ConferenceSystem logs are important data to detect system faults and diagnose root causes of them in a large scale network system. However, due to a huge amount and wide diversity of logs, it is not easy and time consuming for network operators. This paper ...
Detecting Anomalies with : Novel Scores, Architectures, and Settings
Foundations of Intelligent SystemsAbstractis a recently introduced algorithm for unsupervised anomaly detection which enhances latent space-based neural methods, namely (Variational) Autoencoders, GANomaly and ANOGan architectures. The main idea behind it is to exploit both the ...
A Graph Database-Based Approach to Analyze Network Log Files
Network and System SecurityAbstractNetwork log files from different sources often need to be analyzed in order to facilitate a more accurate assessment of the cyber threat severity. For example, using command line tools, any log file can be reviewed only in isolation. While using a ...
Comments