Distributed Cloud Monitoring Platform Based on Log In-Sight

Log management plays an essential role in identifying problems and troubleshoot problems in a distributed system. However, when we conducted log analysis on big data cluster, Kubernetes cluster and Ai capability cluster, we found it was difficult to find a Distributed cloud monitoring platform that met our requirements. So, we propose a Distributed cloud monitoring platform based on log insight, which can be used to achieve unified log insight of big data clusters, K8s clusters, and Ai capability clusters. At the same time, through this system, Developers can intuitively monitor and analyze the business system data and cluster operation monitoring data. Once there is a problem in the log, it will immediately alert, locate, display, and track the message. This system is helpful to improve the readability of log information to administrators, In the process of data collection, Filebeat and Metricbeat will be combined to collect data, therefore, the system can not only collect ordinary log data but also support to collect the indicator data of each famous mature system (Such as operating system, Memcached, Mysql, Docker, Kafka, etc.). Besides, the system will monitor and manage the status of cluster nodes through BeatWatcher. Finally, we develop the system and verify its feasibility and performance by simulation.