Identifying botnets by capturing group activities in DNS traffic

Abstract

Botnets have become the main vehicle to conduct online crimes such as DDoS, spam, phishing and identity theft. Even though numerous efforts have been directed towards detection of botnets, evolving evasion techniques easily thwart detection. Moreover, existing approaches can be overwhelmed by the large amount of data needed to be analyzed. In this paper, we propose a light-weight mechanism to detect botnets using their fundamental characteristics, i.e., group activity. The proposed mechanism, referred to as BotGAD (botnet group activity detector) needs a small amount of data from DNS traffic to detect botnet, not all network traffic content or known signatures. BotGAD can detect botnets from a large-scale network in real-time even though the botnet performs encrypted communications. Moreover, BotGAD can detect botnets that adopt recent evasion techniques. We evaluate BotGAD using multiple DNS traces collected from different sources including a campus network and large ISP networks. The evaluation shows that BotGAD can automatically detect botnets while providing real-time monitoring in large scale networks.

Introduction

A botnet is a network of computers compromised by malicious software. The botnet is operated by a criminal entity to perform Internet attacks, such as identity theft, spam distribution, and DDoS attack. All of these infected hosts are unwilling victims, performing malicious tasks unbeknownst to their owners.

Researchers have focused on bot traffic detection using incidental traits of prevalent bots. However, the detection approaches can be quickly overcome by evasion techniques. Moreover, some approaches need to run with an overwhelming amount of data, which becomes ineffective for high speed networks.

We have proposed a botnet detection mechanism using a fundamental property of botnets [1], [2]. We focused on an underlying common association among infected hosts, command and control (C&C) servers and victims, and found that the botnet generally acts as a coordinated group. Using this “group activity” property, it is possible to detect unknown botnets, irrespective of their communication protocol and structure. BotGAD detects botnets using DNS traffic, since it is possible to capture botnet group activities by monitoring the DNS traffic and monitoring DNS traffic has less overhead than monitoring the entire network traffic. Moreover, DNS monitoring enables botnet detection at their early stages, since botnet DNS traffic is often sent prior to performing attacks.

However, our previous mechanism has three limitations as follows:

• The mechanism may generate false negatives, when a set of infected hosts in a botnet is changed frequently. For example, a part of a botnet can appear only for a short time because they are removed by users or temporally deactivated. The changes in the botnet can decrease detection accuracy of our previous mechanism.

• The previous mechanism is too sensitive against detection parameters. For example, if a time window parameter of the previous mechanism is misconfigured, the mechanism can generate a significant number of false positives or false negatives.

• The mechanism cannot detect recently introduced evasive botnets that utilize a domain generation algorithm for C&C. For example, Kraken/Bobax [3], Srizbi [4], Torpig [5] and Conficker [6] use the domain generation algorithm (DGA) to evade detection.

To overcome the limitations, we improve the mechanism by applying three methods: error correction, cluster analysis, and hypothesis test.

• Error correction. We develop the error correction method to alleviate errors caused when analyzing group activities. Error correction can decrease false alarms caused by unexpected changes in a botnet or by misconfigured detection parameters. We devise column filtering and row filtering operations to correct the errors.

• Cluster analysis. We develop a clustering method using unsupervised machine learning to detect a set of correlated botnets. Several features are devised to classify correlated clusters. Each cluster is analyzed to detect botnet clusters.

• Hypothesis test. We adopt Sequential Probability Ratio Testing (SPRT) [7], as a hypothesis test for sequential analysis, where a decision is made within a small number of rounds with bounded false alarm rates. The SPRT method guarantees a higher level of confidence to make decisions than simple threshold based detection used in our previous mechanism.

The three methods are added in BotGAD to enhance accuracy and robustness against evasions.

We evaluate BotGAD using real-life DNS traces collected from several networks, such as a campus network and a large ISP network. BotGAD can report hundreds of botnet domains and correlated botnet domain clusters. It takes only a few minutes to analyze an hour’s DNS trace of a large ISP network. The evaluation shows BotGAD can automatically detect botnets in real-time, even though they apply evasion techniques, such as the DGA algorithm.

The remainder of this paper is organized as follows. Section 2 reviews related work. We describe the botnet group activity, detection algorithms and a framework of BotGAD in Section 3. We evaluate BotGAD performance in Section 4 and analyze results. We also discuss possible evasion techniques in Section 4. We draw conclusions in Section 5.