Elsevier

Computers & Security

Volume 45, September 2014, Pages 100-123
Computers & Security

An empirical comparison of botnet detection methods

https://doi.org/10.1016/j.cose.2014.05.011Get rights and content

Highlights

  • A comparison of three botnet detection methods using a real dataset.

  • A new, large and public dataset with background, normal and botnet labels.

  • A new performance metric for comparing botnet detection methods in real networks.

  • An analysis and insight view of the impact of botnet activities on the methods.

  • Each method is best for different botnet phases. The keys: a dataset and a metric.

Abstract

The results of botnet detection methods are usually presented without any comparison. Although it is generally accepted that more comparisons with third-party methods may help to improve the area, few papers could do it. Among the factors that prevent a comparison are the difficulties to share a dataset, the lack of a good dataset, the absence of a proper description of the methods and the lack of a comparison methodology. This paper compares the output of three different botnet detection methods by executing them over a new, real, labeled and large botnet dataset. This dataset includes botnet, normal and background traffic. The results of our two methods (BClus and CAMNEP) and BotHunter were compared using a methodology and a novel error metric designed for botnet detections methods. We conclude that comparing methods indeed helps to better estimate how good the methods are, to improve the algorithms, to build better datasets and to build a comparison methodology.

Introduction

It is difficult to estimate how much a new botnet detection method improves the current results in the area. It may be done by comparing the new results with other methods, but this has already been proven hard to accomplish (Aviv and Haeberlen, 2011). Among the factors that prevent these comparisons are: the absence of proper documentation of the methods (Tavallaee et al., 2010), the lack of a common, labeled and good botnet dataset (Rossow et al., 2012), the lack of a comparison methodology (Aviv and Haeberlen, 2011) and the lack of a suitable error metric (Salgarelli et al., 2007).

Although the comparison of methods can greatly help to improve the botnet detection area, few proposals made such a comparison (García et al., 2013). As far as we know, only three papers (Wurzinger et al., 2010, Zhao et al., 2013, Li et al., 2010) made the effort so far.

Obtaining a good dataset for comparisons is difficult. Currently, most detection proposals tend to create their own botnet datasets to evaluate their methods. However, these datasets are difficult to create (Lu et al., 2009) and usually end up being suboptimal (Shiravi et al., 2012), i.e. they lack some important features, such as ground-truth labels, heterogeneity or real-world traffic. These custom datasets are often difficult to use for comparison with other methods. This is because each method is usually focused on different properties of the dataset. The problem is to find a good, common and public dataset that can be read by all methods and satisfy all the constrains.

The difficultly to compare detection methods goes beyond the dataset. The lack of good descriptions of the methods and error metrics contribute to the problem. As stated by Rossow et al. (2012), the error metrics used on most papers are usually non-homogeneous. They tend to use different error metrics and different definitions of error. Moreover, the most common error metrics, e.g. FPR, seems to be not enough to compare botnet detection methods. The classic error metrics were defined from a statistical point of view and they fail to address the detection needs of a network administrator.

The goal of this paper is to compare three botnet detection methods using a simple and reproducible methodology, a good dataset and a new error metric. The contributions of our paper are:

  • A deep comparison of three detection methods. Our own algorithms, CAMNEP and BClus, and the third-party algorithm BotHunter (Gu et al., 2007).

  • A simple methodology for comparing botnet detection methods along with the corresponding public tool for reproducing the methodology.

  • A new error metric designed for comparing botnet detection methods.

  • A new, large, labeled and real botnet dataset that includes botnet, normal and background data.

We conclude that the comparison of different botnet detection methods with other proposals is highly beneficial for the botnet research community because it helps to objectively assess the methods and improve the techniques. Also, that the use of a good botnet dataset is paramount for the comparison.

The rest of the paper is organized as follows. Section 2 shows previous work in the area. Section 3 describes the CAMNEP detection method. Section 4 shows the BClus botnet detection method. Section 5 describes the BotHunter method. Section 6 describes the dataset and its features. Section 7 describes the comparison methodology, the public tool and the new error metric. Section 8 shows the results and compares the methods and Section 9 presents our conclusions.

Section snippets

Previous work

The comparison of detection methods is usually considered a difficult task. In the case of botnets it is also related to the creation of a new dataset. The next Subsections describe the previous work in the area of comparison of methods and the area of creation of datasets.

The CAMNEP detection method

The Cooperative Adaptive Mechanism for NEtwork Protection (CAMNEP) (Rehak et al., 2009) is a Network Behavior Analysis system (Scarfone and Mell, 2007) that consists of various state-of-the-art anomaly detection methods. The system models the normal behavior of the network and/or individual users behaviors and labels deviations from normal behaviors as anomalous.

The BClus detection method

The BClus method is a behavioral-based botnet detection approach. It creates models of known botnet behavior and uses them to detect similar traffic on the network. It is not an anomaly detection method.

The purpose of the method is to cluster the traffic sent by each IP address and to recognize which clusters have a behavior similar to the botnet traffic. A basic schema of the BClus method is:

  • 1.

    Separate the NetFlows in time windows.

  • 2.

    Aggregate the NetFlows by source IP address.

  • 3.

    Cluster the

The BotHunter Method

The BotHunter method was proposed by Gu et al. (2007) to detect the infection and coordination dialog of botnets by matching a state-based infection sequence model. It consists of a correlation engine that aims at detecting specific stages of the malware infection process, such as inbound scanning, exploit usage, egg downloading, outbound bot coordination dialog and outbound attack propagation.

It uses an adapted version of the Snort IDS12 with two proprietary plugin-ins,

Creation of the dataset

In order to compare the methods, a good dataset is needed. According to (Sperotto et al., 2009, Shiravi et al., 2012), a good dataset should be representative of the network were the algorithms are going to be used. This means that it should have botnet, normal and background labeled data, that the balance of the dataset should be like in a real network (usually the percentage of botnet data is small), and that it should be representative of the type of behaviors seen on the network. The

Comparison methodology and new error metric

To compare several detection methods it is necessary to have a methodology, so the comparisons can be repeated and extended. For this purpose we created a simple methodology and a new error metric. The methodology may be used by other researchers to add the results of their methods and obtain a new comparisons. Section 7.1 presents the methodology and Section 7.2 presents the error metric.

Comparison of the results of the detection methods

The three detection methods were executed on each of the five testing datasets described in Section 6.2.2. Each method added its flow predictions to each dataset file, so there are five files to compare using the methodology described in Section 7.

To better understand the implications of comparing these results, the following baseline algorithms were added: the AllPositive algorithm, that always predicts Botnet, the AllNegative algorithm that always predicts Normal and the AllBackground

Conclusions

We conclude that our comparison of detection methods using a real dataset greatly helped to improve our research. It showed us how and why the methods were not optimal, which botnet behaviors were not being detected and how the dataset should be improved. Also, it show us the need for a comparison methodology and a proper error metric.

We also conclude, as it was recommended by Aviv and Haeberlen (2011), that a join effort to create a comparison platform of detection methods could greatly

Acknowledgment

This work was supported by the project of the Czech Ministry of Interior No. VG20122014079.

Sebastián García is a PhD student in UNICEN University (Argentina) and a researcher in the ATG group at the Czech Technical University. He is also a research fellow at the National Scientific and Technical Research Council of Argentina (CONICET) and a teacher in the UFASTA University. His research interests include network-based botnet behavior detection, anomaly detection, penetration testing, honeypots, malware detection, keystroke dynamics and machine learning. His recent projects focus on

References (49)

  • A. Dainotti et al.

    Analysis of a/0 stealth scan from a botnet

  • L. Ertoz et al.

    Minds-minnesota intrusion detection system

  • R. Fontugne et al.

    Mawilab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking

  • S. García

    Malware capture facility project

    (2013)
  • S. García

    Botnet detectors comparer

    (2014)
  • S. García et al.

    Botnet behavior detection using network synchronism

  • S. García et al.

    Survey on network-based botnet detection methods

    J Secur Commun Networks

    (2013)
  • G. Gu et al.

    Bothunter: detecting malware infection through ids-driven dialog correlation

  • G. Gu et al.

    BotSniffer: detecting botnet command and control channels in network traffic

  • M. Hall et al.

    The WEKA data mining software: an update

    ACM SIGKDD Explor Newsl

    (2009)
  • A. Hegna

    Visualizing spatial and temporal dynamics of a class of IRC-based botnets

    (2010)
  • Jacobson V., Leres C., McCanne S. (1997). tcpdump/libpcap www.tcpdump.org Lawrence Berkeley...
  • S. Kotsiantis et al.

    Handling imbalanced datasets : a review

    J GESTS Int Transactions Comput Sci Eng

    (2006)
  • A. Lakhina et al.

    Diagnosing network-wide traffic anomalies

    ACM SIGCOMM Comput Commun Rev

    (2004)
  • Cited by (672)

    View all citing articles on Scopus

    Sebastián García is a PhD student in UNICEN University (Argentina) and a researcher in the ATG group at the Czech Technical University. He is also a research fellow at the National Scientific and Technical Research Council of Argentina (CONICET) and a teacher in the UFASTA University. His research interests include network-based botnet behavior detection, anomaly detection, penetration testing, honeypots, malware detection, keystroke dynamics and machine learning. His recent projects focus on using unsupervised and semi-supervised machine learning techniques to detect botnets on large networks based on their behavioral models.

    Martin Grill holds master degree in Software development at the Faculty of Nuclear Sciences and Physical Engineering of the Czech Technical University in Prague. At the present time he is a member of the Agent Technology Center, a researcher at CESNet, and a PhD student at the Department of Cybernetics of Czech Technical University in Prague.

    Jan Stiborek holds master degree in Software development at Faculty of Nuclear Sciences and Physical Engineering of the Czech Technical University in Prague. At the present time, he is pursuing PhD degree in Artificial Intelligence and Biocybernetics at Department of Cybernetics, FEE CTU. His current profesional interests focus on network security, network simulation and autonomous adaptation of intrusion detection systems.

    Alejandro Zunino (http://www.exa.unicen.edu.ar/∼azunino) received a Ph.D. degree in Computer Science from the National University of the Center of Buenos Aires (UNICEN), in 2003, and his M.Sc. in Systems Engineering in 2000. He is a full Adjunct Professor at UNICEN, member of the ISISTAN Research Institute and Independent Researcher of the National Scientific and Technical Research Council (CONICET). His research areas are Distributed Computing and Software Engineering. Contact him at [email protected].

    View full text