Top

Published in:

2020 | OriginalPaper | Chapter

Leveraging Anomaly Detection for Proactive Application Monitoring

Author : Shyam Zacharia

Published in: Artificial Intelligence XXXVII

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Anomaly detection is one of the popular research fields in Machine Learning. Also, this is one of the key techniques in system and application monitoring in Industry. Anomaly detection comprises of outlier detection and identifying novelty from the data - it is a process to understand the deviation of an observation from existing observations [12] and identifying the new observations. Carrying out anomaly detection in an enterprise application is a challenge as there are complex processes to gather and analyze functional and non-functional logs of unlabeled data. In this paper we are proposing an unsupervised learning process with log featurization incorporating time window to detect outliers and novel errors from enterprise application logs.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Using Sentence Embedding for Cross-Language Plagiarism Detection

next chapter Using Active Learning to Understand the Videoconference Experience: A Case Study

Wang, H., Bah, M., Hammad, M.: Progress in outlier detection techniques: a survey. IEEE Access. 7, 107964–108000 (2019). https://doi.org/10.1109/access.2019.2932769CrossRef

Akoglu, L., Tong, H., Vreeken, J., Faloutsos, C.: Fast and reliable anomaly detection in categorical data. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management - CIKM ‘12 (2012). https://doi.org/10.1145/2396761.2396816

He, S., Zhu, J., He, P., Lyu, M.: Experience report: system log analysis for anomaly detection. In: 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE) (2016). https://doi.org/10.1109/issre.2016.21

Baur, C., Wiestler, B., Albarqouni, S., Navab, N.: Deep autoencoding models for unsupervised anomaly segmentation in brain MR images. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11383, pp. 161–169. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11723-8_16CrossRef

Breunig, M., Kriegel, H., Ng, R., Sander, J.: LOF. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data - SIGMOD ‘00 (2000)

Gensim Home Page. https://radimrehurek.com/gensim/models/word2vec.html. Accessed 25 Jun 2020

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles - SOSP ‘09 (2009)

Yamanishi, K., Maruyama, Y.: Dynamic syslog mining for network failure monitoring. In: Proceeding of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining - KDD ‘05 (2005)

10.

Nagaraj, K., Killian, C., Neville, J.: Structured comparative analysis of systems logs to diagnose performance problems. NSDI, pp. 353–366 (2012)

11.

Ghanbari, S., Hashemi, A.B., Amza, C.: Stage-aware anomaly detection through tracking log points. In: Proceedings of the 15th International Middleware Conference, pp. 253–264 (2014). https://doi.org/10.1145/2663165.2663319

12.

Hawkins, D.: Identification of Outliers. Chapman and Hall, London (1980)CrossRef

13.

Barnett, V., Lewis, T.: Outliers in Statistical Data. Wiley, New York (1994)MATH

14.

Zhao, Y., Nasrullah, Z., Li, Z.: PyOD: a python toolbox for scalable outlier detection. J. Mach. Learn. Res. 20(96), 1–7 (2019)

15.

Schölkopf, B., Platt, J., Shawe-Taylor, J., Smola, A., Williamson, R.: Estimating the support of a high-dimensional distribution. Neural Comput. 13, 1443–1471 (2001)CrossRef

16.

Kriegel, H.P., Schubert, M., Zimek, A.: Angle-based outlier detection in high-dimensional data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 444–452 (2008)

17.

He, Z., Xu, X., Deng, S.: Discovering cluster-based local outliers. Pattern Recogn. Lett. 24, 1641–1650 (2003)CrossRef

18.

Lazarevic, A., Kumar, V.: Feature bagging for outlier detection. In: Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, 157–166 (2005)

19.

Goldstein, M., Dengel, A.: Histogram-based outlier score (hbos): aa fast unsupervised anomaly detection algorithm. KI-2012: Poster and Demo Track, pp. 59–63 (2012)

20.

Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 413–422 (2008). https://doi.org/10.1109/icdm.2008.17

21.

Angiulli, F., Pizzuti, C.: Fast outlier detection in high dimensional spaces. In: European Conference on Principles of Data Mining and Knowledge Discovery, pp. 15–27 (2002)

22.

Hardin, J., Rocke, D.: Outlier detection in the multiple cluster setting using the minimum covariance determinant estimator. Comput. Stat. Data Anal. 44, 625–638 (2004)MathSciNetCrossRef

23.

Shyu, M.L., Chen, S.C., Sarinnapakorn, K. and Chang, L.: A novel anomaly detection scheme based on principal component classifier. Miami Univ Coral Gables Fl Dept of Electrical and Computer Engineering (2003)

24.

Hanley, J., McNeil, B.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29–36 (1982)CrossRef

25.

Craswell, N.: Precision at n. Encyclopedia of Database Systems, pp. 2127–2128 (2009)

Title: Leveraging Anomaly Detection for Proactive Application Monitoring
Author: Shyam Zacharia
Publisher: Springer International Publishing
Book: Artificial Intelligence XXXVII
Print ISBN: 978-3-030-63798-9

Electronic ISBN: 978-3-030-63799-6

Copyright Year: 2020
DOI: https://doi.org/10.1007/978-3-030-63799-6_29

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"