Dlog: diagnosing router events with syslogs for anomaly detection

Router systems are notoriously difficult to understand or diagnose for their closure and heterogeneity. A common way of gaining insight into the router system and detecting the anomaly behaviors is to inspect the router syslogs. Unfortunately, syslogs are difficult to inspect because they are large-scale, unstructured and various in different vendors and services. Besides, they are too low-level to be directly used in anomaly detection. Prevalent approaches to understanding syslogs focus on simple keyword search (such as error and exception) of logs that may be associated with the failures. Such an approach is time consuming and error prone. In this paper, we present Dlog which can automatically transform and compress such low-level and minimally structured syslog messages into meaningful and prioritized high-level network events that can be used in anomaly detection. Dlog has two main steps: the first is the training process that learns the features of the normal and abnormal events; the second is anomaly detection and classification which can detect the anomalous events and provide the network operators with specific attack modes. We have applied our approach in a university network which contains Cisco, Huawei and Dlink routers for 5 months. We aligned our experiment with a former work as a baseline for comparison. Dlog is 23% faster in log template extraction and has improved the accuracy rate in template extraction 2 times higher than the former work. Besides, we can achieve 96% precision rate in anomaly detection and provide users with the attack modes in seven clusters.