A Logging Overhead Optimization Method Based on Anomaly Detection Model

Logs play an important role in system anomaly detection. However, in today’s large-scale software development and production, the cost of logging is non-negligible, and intensive logging in actual production processes will generate a large amount of redundant logs which are useless for anomaly detection. However, the current method of solving related problems is not ideal, and it is only applied when the overhead of the logging has affected the quality of service. Therefore, this paper proposes a method for optimizing logging records for this problem. Under a given budget (defined as the maximum volume of logs allowed to be output in a time interval), using an anomaly detection model based on deep learning and a two-phase filtering mechanism, the method determines whether to log according to the utility score of the log for anomaly detection to save useful logs and discard less useful logs during the system running process. The experimental results show that the proposed method alleviates the logging overhead problem without reducing the logging effectiveness.