Anzeige
Erschienen in:
2017 | OriginalPaper | Buchkapitel
Hive-Based Anomaly Detection in Hadoop Log Data Management
verfasst von : Siwoon Son, Myeong-Seon Gil, Seokwoo Yang, Yang-Sae Moon
Erschienen in: Advances in Computer Science and Ubiquitous Computing
Verlag: Springer Singapore
Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.
Wählen Sie Textabschnitte aus um mit Künstlicher Intelligenz passenden Patente zu finden. powered by
Markieren Sie Textabschnitte, um KI-gestützt weitere passende Inhalte zu finden. powered by
Abstract
In this paper, we address how to manage and analyze a large volume of log data, which have been difficult to be handled in the traditional computing environment. To handle a large volume of Hadoop log data, which rapidly occur in multiple servers, we present new data storage architecture to efficiently analyze those big log data through Apache Hive. We then design and implement a simple but efficient anomaly detection method, which identifies abnormal status of servers from log data, based on moving average and 3-sigma techniques. We also show effectiveness of the proposed detection method by demonstrating that it properly detects anomalies from Hadoop log data.