Automatic Malware Analysis

Malicious software (i.e., malware) has become a severe threat to interconnected computer systems for decades and has caused billions of dollars damages each year. A large volume of new malware samples are discovered daily. Even worse, malware is rapidly evolving to be more sophisticated and evasive to strike against current malware analysis and defense systems. The work described in this book takes a root-cause oriented approach to the problem of automatic malware analysis. In this approach, we aim to capture the intrinsic natures of malicious behaviors, rather than the external symptoms of existing attacks. We propose a new architecture for binary code analysis, which is called whole-system out-of-the-box fine-grained dynamic binary analysis, to address the common challenges in malware detection and analysis. To realize this architecture, we build a unified and extensible analysis platform, code-named TEMU. We propose a core technique for fine-grained dynamic binary analysis, called layered annotative execution, and implement this technique in TEMU. Then on the basis of TEMU, we have proposed and built a series of novel techniques for automatic malware detection and analysis. We have developed Renovo, Panorama, HookFinder, and MineSweeper, for detecting and analyzing various aspects of malware. These techniques capture intrinsic characteristics of malware and thus are well suited for dealing with new malware samples and attack mechanisms.

Dynamic binary analysis has demonstrated its strength in solving a wide-spectrum of computer security problems, especially malware analysis. An extensible platform for dynamic binary analysis provides a foundation for solving these problems. To enable a variety of applications, we explore a unique design space. We aim to provide a whole-system view, take an external approach, facilitate fine-grained instrumentation, and have sufficient efficiency. These design goals bring about a new architecture, namely whole-system out-of-the-box fine-grained dynamic binary analysis. To further facilitate fine-grained dynamic binary analysis, we propose layered annotative execution as a core technique, which incorporates shadow flag analysis, taint analysis, and symbolic execution. We have implemented this new architecture and the core technique in an analysis platform called TEMU. Because of its extensibility and versatility, TEMU serves as the foundation for numerous malware analysis techniques.

As reverse engineering becomes a prevalent technique to analyze malware, malware writers leverage various anti-reverse engineering techniques to hide their code. One technique commonly used is code packing as packed executables hinder code analysis. While this problem has been previously researched, the existing solutions are either unable to handle novel samples, or vulnerable to various evasion techniques. In this chapter, we propose a fully dynamic approach that captures an intrinsic nature of hidden code execution that the original code should be present in memory and executed at some point at run-time. Thus, this approach monitors program execution and memory writes at run-time, determines if the code under execution is newly generated, and then extracts the hidden code of the executable. To demonstrate its effectiveness, we implement a system, Renovo, and evaluate it with a large number of real-world malware samples. The experiments show that Renovo is accurate compared to previous work, yet practical in terms of performance.

Malicious programs spy on users’ behavior and compromise their privacy. Even software from reputable vendors, such as Google Desktop and Sony DRM media player, may perform undesirable actions. Unfortunately, existing techniques for detecting malware and analyzing unknown code samples are insufficient and have significant shortcomings. We observe that malicious information access and processing behavior is the fundamental trait of numerous malware categories breaching users’ privacy (including keyloggers, password thieves, network sniffers, stealth backdoors, spyware and rootkits), which separates these malicious applications from benign software. We propose a system, Panorama, to detect and analyze malware by capturing this fundamental trait. In our extensive experiments, Panorama successfully detected all the malware samples and had very few false positives. Furthermore, by using Google Desktop as a case study, we show that our system can accurately capture its information access and processing behavior, and we can confirm that it does send back sensitive information to remote servers in certain settings. We believe that a system such as Panorama will offer indispensable assistance to code analysts and malware researchers by enabling them to quickly comprehend the behavior and innerworkings of an unknown sample.

Installing various hooks into the victim system is an important attacking strategy employed by malware, including spyware, rootkits, stealth backdoors, and others. In order to defeat existing hook detectors, malware writers keep exploring new hooking mechanisms. However, the current malware analysis procedure is painstaking, mostly manual and error-prone. In this chapter, we propose the first systematic approach for automatically identifying hooks and extracting hooking mechanisms. We propose a unified approach, fine-grained impact analysis, to identify malware hooking behaviors. Our approach does not rely on any prior knowledge of hooking mechanisms, and thus can identify novel hooking mechanisms. Moreover, we devise a method using semantics-aware impact dependency analysis to provide a succinct and intuitive graph representation to illustrate hooking mechanisms. We have developed a prototype, HookFinder, and conducted extensive experiments using representative malware samples from various categories. We have demonstrated that HookFinder can correctly identify the hooking behaviors of all samples, and provide accurate insights about their hooking mechanisms.

Malware often contains hidden behavior which is only activated when properly triggered. Well known examples include: the MyDoom worm which DDoS’s on particular dates, keyloggers which only log keystrokes for particular sites, and DDoS zombies which are only activated when given the proper command. We call such behavior trigger-based behavior. Currently, trigger-based behavior analysis is often performed in a tedious, manual fashion. Providing even a small amount of assistance would greatly assist and speedup the analysis. In this chapter, we propose that automatic analysis of trigger-based behavior in malware is possible. In particular, we design an approach for automatic trigger-based behavior detection and analysis using dynamic binary instrumentation and mixed concrete and symbolic execution. Our approach shows that in many cases we can: (1) detect the existence of trigger-based behavior, (2) find the conditions that trigger such hidden behavior, and (3) find inputs that satisfy those conditions, allowing us to observe the triggered malicious behavior in a controlled environment. We have implemented MineSweeper, a system utilizing this approach. In our experiments, MineSweeper has successfully identified trigger-based behavior in real-world malware. Although there are many challenges presented by automatic trigger-based behavior detection, MineSweeper shows us that such automatic analysis is possible and encourages future work in this area.

In this book, we sought to capture the intrinsic natures in malicious behaviors, in order to build more effective automatic malware analysis systems. We proposed TEMU as a dynamic binary analysis platform. Then on top of TEMU, we proposed and built a series of novel techniques for automatic malware analysis, including Renovo, Panorama, HookFinder, and MineSweeper, for detecting and analyzing various aspects of malware. Since these techniques capture intrinsic characteristics of malware, they are well suited for dealing with new malware samples and attack mechanisms. We also systematically discussed several fundamental limitations in our proposed techniques.More concretely, we pointed out that although our analysis platform is better suited for analyzing malicious code than the other conventional ones (e.g., debugger and disassembler), malware authors may still find ways to detect and evade it. Moreover, an open problem for dynamic analysis lies in its limited test coverage. Finally, as a core analysis technique, dynamic taint analysis has several limitations, including taint explosion and implicit information flow. Such discussions shed light on future directions for automatic malware analysis.