An empirical comparison of botnet detection methods
Introduction
It is difficult to estimate how much a new botnet detection method improves the current results in the area. It may be done by comparing the new results with other methods, but this has already been proven hard to accomplish (Aviv and Haeberlen, 2011). Among the factors that prevent these comparisons are: the absence of proper documentation of the methods (Tavallaee et al., 2010), the lack of a common, labeled and good botnet dataset (Rossow et al., 2012), the lack of a comparison methodology (Aviv and Haeberlen, 2011) and the lack of a suitable error metric (Salgarelli et al., 2007).
Although the comparison of methods can greatly help to improve the botnet detection area, few proposals made such a comparison (García et al., 2013). As far as we know, only three papers (Wurzinger et al., 2010, Zhao et al., 2013, Li et al., 2010) made the effort so far.
Obtaining a good dataset for comparisons is difficult. Currently, most detection proposals tend to create their own botnet datasets to evaluate their methods. However, these datasets are difficult to create (Lu et al., 2009) and usually end up being suboptimal (Shiravi et al., 2012), i.e. they lack some important features, such as ground-truth labels, heterogeneity or real-world traffic. These custom datasets are often difficult to use for comparison with other methods. This is because each method is usually focused on different properties of the dataset. The problem is to find a good, common and public dataset that can be read by all methods and satisfy all the constrains.
The difficultly to compare detection methods goes beyond the dataset. The lack of good descriptions of the methods and error metrics contribute to the problem. As stated by Rossow et al. (2012), the error metrics used on most papers are usually non-homogeneous. They tend to use different error metrics and different definitions of error. Moreover, the most common error metrics, e.g. FPR, seems to be not enough to compare botnet detection methods. The classic error metrics were defined from a statistical point of view and they fail to address the detection needs of a network administrator.
The goal of this paper is to compare three botnet detection methods using a simple and reproducible methodology, a good dataset and a new error metric. The contributions of our paper are:
- •
A deep comparison of three detection methods. Our own algorithms, CAMNEP and BClus, and the third-party algorithm BotHunter (Gu et al., 2007).
- •
A simple methodology for comparing botnet detection methods along with the corresponding public tool for reproducing the methodology.
- •
A new error metric designed for comparing botnet detection methods.
- •
A new, large, labeled and real botnet dataset that includes botnet, normal and background data.
We conclude that the comparison of different botnet detection methods with other proposals is highly beneficial for the botnet research community because it helps to objectively assess the methods and improve the techniques. Also, that the use of a good botnet dataset is paramount for the comparison.
The rest of the paper is organized as follows. Section 2 shows previous work in the area. Section 3 describes the CAMNEP detection method. Section 4 shows the BClus botnet detection method. Section 5 describes the BotHunter method. Section 6 describes the dataset and its features. Section 7 describes the comparison methodology, the public tool and the new error metric. Section 8 shows the results and compares the methods and Section 9 presents our conclusions.
Section snippets
Previous work
The comparison of detection methods is usually considered a difficult task. In the case of botnets it is also related to the creation of a new dataset. The next Subsections describe the previous work in the area of comparison of methods and the area of creation of datasets.
The CAMNEP detection method
The Cooperative Adaptive Mechanism for NEtwork Protection (CAMNEP) (Rehak et al., 2009) is a Network Behavior Analysis system (Scarfone and Mell, 2007) that consists of various state-of-the-art anomaly detection methods. The system models the normal behavior of the network and/or individual users behaviors and labels deviations from normal behaviors as anomalous.
The BClus detection method
The BClus method is a behavioral-based botnet detection approach. It creates models of known botnet behavior and uses them to detect similar traffic on the network. It is not an anomaly detection method.
The purpose of the method is to cluster the traffic sent by each IP address and to recognize which clusters have a behavior similar to the botnet traffic. A basic schema of the BClus method is:
- 1.
Separate the NetFlows in time windows.
- 2.
Aggregate the NetFlows by source IP address.
- 3.
Cluster the
The BotHunter Method
The BotHunter method was proposed by Gu et al. (2007) to detect the infection and coordination dialog of botnets by matching a state-based infection sequence model. It consists of a correlation engine that aims at detecting specific stages of the malware infection process, such as inbound scanning, exploit usage, egg downloading, outbound bot coordination dialog and outbound attack propagation.
It uses an adapted version of the Snort IDS12 with two proprietary plugin-ins,
Creation of the dataset
In order to compare the methods, a good dataset is needed. According to (Sperotto et al., 2009, Shiravi et al., 2012), a good dataset should be representative of the network were the algorithms are going to be used. This means that it should have botnet, normal and background labeled data, that the balance of the dataset should be like in a real network (usually the percentage of botnet data is small), and that it should be representative of the type of behaviors seen on the network. The
Comparison methodology and new error metric
To compare several detection methods it is necessary to have a methodology, so the comparisons can be repeated and extended. For this purpose we created a simple methodology and a new error metric. The methodology may be used by other researchers to add the results of their methods and obtain a new comparisons. Section 7.1 presents the methodology and Section 7.2 presents the error metric.
Comparison of the results of the detection methods
The three detection methods were executed on each of the five testing datasets described in Section 6.2.2. Each method added its flow predictions to each dataset file, so there are five files to compare using the methodology described in Section 7.
To better understand the implications of comparing these results, the following baseline algorithms were added: the AllPositive algorithm, that always predicts Botnet, the AllNegative algorithm that always predicts Normal and the AllBackground
Conclusions
We conclude that our comparison of detection methods using a real dataset greatly helped to improve our research. It showed us how and why the methods were not optimal, which botnet behaviors were not being detected and how the dataset should be improved. Also, it show us the need for a comparison methodology and a proper error metric.
We also conclude, as it was recommended by Aviv and Haeberlen (2011), that a join effort to create a comparison platform of detection methods could greatly
Acknowledgment
This work was supported by the project of the Czech Ministry of Interior No. VG20122014079.
Sebastián García is a PhD student in UNICEN University (Argentina) and a researcher in the ATG group at the Czech Technical University. He is also a research fellow at the National Scientific and Technical Research Council of Argentina (CONICET) and a teacher in the UFASTA University. His research interests include network-based botnet behavior detection, anomaly detection, penetration testing, honeypots, malware detection, keystroke dynamics and machine learning. His recent projects focus on
References (49)
- et al.
Data preprocessing for anomaly based network intrusion detection: a review
J Comput Secur
(2011) - et al.
DDoS attack detection method using cluster analysis
Expert Syst Appl
(2008) - et al.
Toward developing a systematic approach to generate benchmark datasets for intrusion detection
J Comput Secur
(2012) - et al.
Botnet detection based on traffic behavior analysis and flow intervals
J Comput Secur
(2013) Auditing Network Activity
(2013)- et al.
Periodic behavior in botnet command and control channels traffic
- et al.
Challenges in experimenting with botnet detection systems
- et al.
Behavior-based Malware clustering
- et al.
Traffic data repository at the WIDE project
Specification of the IP Flow Information Export (IPFIX) protocol for the exchange of IP traffic flow information
(2008)
Analysis of a/0 stealth scan from a botnet
Minds-minnesota intrusion detection system
Mawilab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking
Malware capture facility project
Botnet detectors comparer
Botnet behavior detection using network synchronism
Survey on network-based botnet detection methods
J Secur Commun Networks
Bothunter: detecting malware infection through ids-driven dialog correlation
BotSniffer: detecting botnet command and control channels in network traffic
The WEKA data mining software: an update
ACM SIGKDD Explor Newsl
Visualizing spatial and temporal dynamics of a class of IRC-based botnets
Handling imbalanced datasets : a review
J GESTS Int Transactions Comput Sci Eng
Diagnosing network-wide traffic anomalies
ACM SIGCOMM Comput Commun Rev
Cited by (672)
Bot-DM: A dual-modal botnet detection method based on the combination of implicit semantic expression and graphical expression
2024, Expert Systems with ApplicationsAn intelligent multi-layer framework with SHAP integration for botnet detection and classification
2024, Computers and SecurityPeerG: A P2P botnet detection method based on representation learning and graph contrastive learning
2024, Computers and SecurityAdversarial examples: A survey of attacks and defenses in deep learning-enabled cybersecurity systems
2024, Expert Systems with ApplicationsGraph-based few-shot incremental learning algorithm for unknown class detection
2024, Applied Soft ComputingBotnet sequential activity detection with hybrid analysis
2024, Egyptian Informatics Journal
Sebastián García is a PhD student in UNICEN University (Argentina) and a researcher in the ATG group at the Czech Technical University. He is also a research fellow at the National Scientific and Technical Research Council of Argentina (CONICET) and a teacher in the UFASTA University. His research interests include network-based botnet behavior detection, anomaly detection, penetration testing, honeypots, malware detection, keystroke dynamics and machine learning. His recent projects focus on using unsupervised and semi-supervised machine learning techniques to detect botnets on large networks based on their behavioral models.
Martin Grill holds master degree in Software development at the Faculty of Nuclear Sciences and Physical Engineering of the Czech Technical University in Prague. At the present time he is a member of the Agent Technology Center, a researcher at CESNet, and a PhD student at the Department of Cybernetics of Czech Technical University in Prague.
Jan Stiborek holds master degree in Software development at Faculty of Nuclear Sciences and Physical Engineering of the Czech Technical University in Prague. At the present time, he is pursuing PhD degree in Artificial Intelligence and Biocybernetics at Department of Cybernetics, FEE CTU. His current profesional interests focus on network security, network simulation and autonomous adaptation of intrusion detection systems.
Alejandro Zunino (http://www.exa.unicen.edu.ar/∼azunino) received a Ph.D. degree in Computer Science from the National University of the Center of Buenos Aires (UNICEN), in 2003, and his M.Sc. in Systems Engineering in 2000. He is a full Adjunct Professor at UNICEN, member of the ISISTAN Research Institute and Independent Researcher of the National Scientific and Technical Research Council (CONICET). His research areas are Distributed Computing and Software Engineering. Contact him at [email protected].