Malicious code detection in android: the role of sequence characteristics and disassembling methods

Android mobile operating system has a market share of 73% for smartphones, and 43% for tablets [1]. Thanks to its open-source feature, the Android operating system offers an affordable and convenient interface for end-users. It also provides a well-established development environment and infrastructure for mobile software developers. Hence, an inevitable consequence is the close attention of malware authors, who target the Android platform by mostly embedding their malicious code into benign applications. Nowadays, malicious code detection, as implemented by antivirus software or anti-malware tools, is a native part of many desktop operating systems. Nevertheless, mobile operating systems are most vulnerable to malware. In 2020, Kaspersky Mobile products and technologies detected a 62.2% annual increase in the installation of malicious packages on Android environment [2]. This growing threat points out the urgent need for anti-malware tools running with high detection accuracies.

In general, malware analysis methods can be grouped under three different categories: static, dynamic, and hybrid analysis methods. The static analysis relies on examining the source code of a program and accompanying metadata without running the binary itself. The dynamic analysis aims to investigate the behavior of the executable by running it. Lastly, the hybrid analysis leverages the combinations of the other two analysis methods. Early anti-malware software employed signature-based protection mechanisms that created traces of known malware from existing malware recorded in constantly updated databases. In these systems, signatures, and traces are devised by utilizing the previously mentioned malware analysis techniques. The aforementioned paradigm suffered from the limitations of the size of the databases, the temporal gap between the detection of malware and the implementation of the signature in the database (cf. zero-day vulnerabilities), and the ease of obfuscation techniques creating versions of the same malware with different signatures. Therefore, malware detection methods that offer adaptive solutions, including machine learning techniques, have replaced signature-based methodologies. Unlike signature-based methods, the methods with machine learning (ML) techniques utilize malware analysis methodologies to perform feature extraction from Android application (apk) files [21, 39]–[42].

A common practice in evaluating ML-based malware detection methods and models, specifically the classification accuracy of binaries as benign or malware, is to conduct bench-marking by reporting accuracy values in contingency tables. This common practice has proved to be an efficient evaluation method, allowing researchers to perform comparative analyses of their methodologies. Nevertheless, besides the multi-parameter nature of learning machines such as deep learning methods, the other factors, like the input unit(as formed by the code structures), the input representation specified by the disassembling tool, and the length of sequences used in language modeling may also have a high impact on the results. However, due to the complex nature of malware detection, those factors and the parameter values relevant to them are usually conceived as operational assumptions rather than being treated as determining factors with a high impact on the results.

The goal of the present study is twofold: We present a methodology for Android malware detection using Long Short-Term Memory (LSTM) by following our previous research [37]. Then, we have a closer look at the factors that have an impact upon the model outcomes. In particular, we focus on the role of the disassembler software since it identifies the representation of the input. Our findings show that the disassembling method influences the processing, thus the model outcomes. Another factor is the form of the input data as specified by the researcher. In particular, a researcher may prefer to use a set of instructions, basic blocks, methods, or classes as the unit of input representation. Our findings reveal how different types of input representation influence the model outcomes.

For model development, we employed a specialized type of Recurrent Neural Network (RNN), namely the LSTM [3]. We handle malware detection from a Natural Language Processing (NLP) perspective. For this purpose, we develop and test the results in various basic units (instruction, basic block, method, and class) generated by the JEB Decompiler [4], the IDA Pro Disassembler [5], and the Apktool [6], which are widely used tools in reverse engineering apk files. More specifically, we reverse-engineered the apk files by the JEB Decompiler, the IDA Pro Disassembler, and the Apktool. First, the benign and malicious apk files were decoded, then the resulting codes were divided into instructions, basic blocks, methods, and classes. Finally, LSTM models were trained on those units with the Dalvik assembly code of malicious and benign apk files.

The following section describes the relevant studies about malware detection on Android by employing machine learning and deep learning techniques. Afterward, we present our proposed methodology, including the approach, the details on datasets, and the LSTM pipeline. We then report the results and present a discussion of the findings within the framework of the literature.

The popularity of Android is of interest to malware developers as well as researchers who develop anti-malware solutions. Thus, in the past two decades, detecting and analyzing Android malware has become an active area of research in cyber security. An indispensable part of the research effort has been integrating machine learning techniques, more recently the adaption of deep learning methodologies for higher accuracy in detection rates, besides the opportunity to detect novel malware types. In this section, we present the relevant work on Android malware detection methods.

In a survey study, Sharma and Rattan conducted a review on Android malware detection [7]. They categorized malware analysis techniques into three groups, namely static, dynamic, and hybrid analysis. The static analysis refers to reviewing an apk file without executing the application, mainly focusing on the manifest and the source code file. In some instances, malware authors obfuscate the apk files to reduce the efficiency of malware detection by increasing the analysis time. Nevertheless, static analysis is a fast and popular technique for malware detection. An alternative to static malware analysis is dynamic analysis. With dynamic analysis, the application is executed, and its behavior is observed. In dynamic analysis, a separate, isolated environment is needed to run the application. Dynamic analysis is a more complex process that may become a resource- and time-consuming. However, it is a widely used and reliable technique. Lastly, hybrid strategies use the combination of static and dynamic analysis to inspect the application by utilizing the two analysis methods together. A significant advantage of static analysis is its lightweight infrastructure and low-cost efficiency. Therefore, in the present study, we used static analysis. We used smali files disassembled by alternative disassembly tools as inputs to the model.

A review of the relevant work shows that various machine learning models and model features have been employed in the static analysis of malicious code. The most common models and learning algorithms include Decision Trees (DT), Support Vector Machines (SVM), Naïve Bayes (NB), Logistic Regression (LR), Ensemble Learning (EL), Online Learning (OL), K-Nearest Neighbor (KNN), and neural networks (e.g., RNN, CNN, MLP, LSTM). In addition, various features have been selected for the models of Android malware detection, including application permissions, API calls, intents, opcode sequences, program graphs (e.g., function call graphs, control flow graphs), and hardware features obtained from Android apk files [8]–[19]. In our present study, we utilized a deep learning model, LSTM, and trained the model with the instruction sequences extracted from disassembled smali files of applications. In other words, we used opcode sequences as the features and LSTM as the learning model.

Numerous studies have employed Natural Language Processing (NLP) Deep Learning models to develop malware detection methods. For instance, Karbab et al. utilized NLP models and proposed a framework that relied on API method call sequences to detect and classify Android malware [24]. They obtained dex files from apk files for the API call extraction by using Python’s zip library. The Dalvik assembly code was obtained from dex files with dexdump. The API call sequences were then extracted by utilizing regular expressions. Those sequences were used to generate semantic vectors with word2vec and then fed to a CNN (Convolutional Neural Network) model to perform detection and classification. The suggested framework achieved a detection accuracy of 96–99%. In a similar study [26], Ma et al. proposed a method called Droidetec, which represented applications in terms of natural language sequences by utilizing the API calls. They used the Androguard [43] tool to retrieve dex files from apk files. After extracting API sequences with some pre-processing, one-hot vector embedding and Skip-Gram model were used to set up the input vector of sequence. The detection was performed by forwarding this input to the LSTM-based detection model. Aside from malware detection, Droidetec adopted an attention mechanism that utilized weight distributions of APIs to localize the potentially malicious code and output a report providing specific information on malicious code segments. Droidetec reached over 97% accuracy for detection and a 91% hit rate for detecting malicious code. Wu et al. [25] employed a similar semantic view for malware detection. Instead of the API calls, the researchers used requested permissions extracted from the Android manifest file AndroidManifest.xml. After the extraction, a threshold mechanism was employed to filter out standard permissions. Then by word2vec, permissions were represented as a one-hot vector fed into the LSTM model for malware detection. A different approach was proposed by Xie et al., which offered an LSTM-based method that applied dynamic analysis [27]. They used system calls as the model features, and they represented the program’s behavior with system-call sequences (viz. the behavioral language in their terminology). The model was tested with both host-based and Android-based datasets and achieved 99% and 95% accuracy, respectively. In another study, the authors used both static and dynamic analysis techniques giving an example of a hybrid method for malware analysis [33]. For the static analysis, they extracted permissions of applications from Android manifest files using the Apktool. The battery, binder, memory, and permissions were collected using emulators and feature collection platforms, then used as the features in the dynamic analysis. The researchers used various combinations of the features on RNN (Recurrent Neural Network) and LSTM models for comparative analysis. Their results showed that the LSTM outperformed RNN, achieving a detection rate of 97% for static analysis and 93% for dynamic analysis. More recently, a malware classification framework, namely ROCKY, performed malicious code detection by extracting API calls, permission, keywords, and tokens from decompiled source code instead of Dalvik bytecode or smali code [28]. It applied lexical analysis of the source code by utilizing sequences of N-tokens that include stop-tokens, feature-tokens, and long-tail tokens. The developers compared ROCKY with other neural network models reported in the previous work. Their findings revealed that ROCKY outperformed other models and features, achieving an accuracy of 97.5%. In contrast to the studies that applied semantic-based techniques using features like API calls and permissions, our models use the smali code of android applications to take advantage of the LSTM model in learning the behavior of the applications.

A further review of the application of the learning methods for malicious code detection reveals that opcode sequences have often been used in malware detection. For instance, McLaughlin et al. proposed a CNN-based malware detection approach [20]. They used the Baksmali [44] tool to extract opcode sequences and proposed a Hierarchical Denoising Network model to deal with the Gradient vanishing problem of LSTM by using encoders. So, the basic idea was to disassemble each apk file into smali files by using Baksmali, and then remove the operands to obtain opcode sequences from each method in all the smali files. Gathering the opcode sequences into a single opcode sequence makes it possible to represent the whole application. Text representations of applications were then fed to the CNN model to perform the detection. The main advantage of the study was that it did not require complex, manual preprocessing for data preparation. Parker et al. conducted a similar study using the same data preparation and disassembler tools [21]. However, they used two different classification methods, namely weighted sequence and Modified Multi-Layer Vector Space models. Their methods did not explicitly target Android malware detection. Their results achieved 79% accuracy. In another study, Li et al. adopted the same data preprocessing approach using a different disassembler, namely Apktool, to collect the smali files [22]. They used k-max pooling models and compared their model with a set of classical machine learning models, such as RF, DT, SVM, Naïve Bayes, and MLP. Their model reached 99% of accuracy and outperformed the others.

Numerous studies have employed features, such as opcode sequences or source code in LSTM models, alongside the previously mentioned research focusing on opcode sequences. For example, Deeprefiner was proposed as a semantic-based, two-layered method for malware detection [29]. Deeprefiner’s first layer performs an initial detection by inputting vector representations of Android manifest files and the other XML files under the resources directory to an MLP model. Then, the unclassified applications are forwarded to the second detection layer for a more thorough inspection. In the second layer, dex bytecode, extracted by the Apktool, is represented as vectors using Skip-Gram modeling, then fed into the LSTM model. The preprocessing of the bytecode involves reducing all dex instructions to fifteen instruction categories without removing information like class names, fields, and methods.

Other studies that used opcode sequences with LSTM models include [31], in which the Apktool was used to obtain opcode sequences. The opcodes were then converted to 16-bit code representations. The model also utilized an opcode sequence encoder to reduce feature dimensions. In [34], Amin et al. adopted a similar approach using opcode sequences and LSTM for malware detection and family attribution. The authors conducted experiments with other neural network architectures. They stated that BiLSTM outperformed other models without the burden of manual preprocessing for the features. A similar study used opcode sequences retrieved from instruction call graphs [23]. The Androguard tool was used for decompiling the applications to obtain all the execution paths. The embedded opcode sequences were then fed into the LSTM model, and the detection was performed by evaluating the features of the graph representation alongside the LSTM. In [32], the researchers reported experiments with CNN and LSTM models applied on two datasets. For the LSTM model, the authors extracted Android bytecode using the Apktool. The bytes were then represented as vectors by one-hot encoding and then fed into the LSTM model. The proposed LSTM architecture achieved 70% accuracy for small datasets and 95.3% accuracy for large datasets. In our study, we used the instructions obtained from smali files without any manual preprocessing. We also included operands besides the opcodes in the instructions to avoid making simplification assumptions. A unique aspect of our study is the investigation of the role of the disassembler tool. We used Apktool, JEB, and IDA Pro to disassemble apk files to obtain smali files and reported the differences. The following section presents the methodology of the study.

In this section, we present the results of the LSTM models trained on the datasets. We used three datasets (JEB, IDA, Apktool) for Dataset 1 and Dataset 2 in our ISM experiments by simply removing the lines irrelevant to the program functionality in the assembly output, such as comment lines and the lines with blanks. We call this first set of datasets the base datasets since we did not change the data format. In the Basic Blocks Sequences Model (BSM) datasets, we performed preprocessing on the base datasets to change the data format. In preprocessing, we put the sequential instructions together by separating them from the basic block endpoints. So, we obtained the datasets that included the same data but in a different format. In the second set of datasets, each sequence consisted of a basic block. In the third set of datasets, we used the base datasets obtained from the three disassemblers and performed preprocessing to change the data format as intended. In the preprocessing phase, we put the instructions in the same methods together by separating them from the method’s beginning and endpoints. Thus, we obtained a new set of datasets that included the same data with the ISM and BSM but in different formats. Those new datasets included sequences consisting of methods called MSM. In the last set of datasets, we worked on the base datasets by performing preprocessing to change the data format. In the preprocessing, we performed similar operations as in ISM and BSM by putting the sequential instructions in the same classes together by separating them from the class beginning points. In those datasets, each sequence consisted of a class and they are called CSM. We trained each model on JEB, IDA, and Apktool datasets using the aforementioned parameter values. Tables 4 and 5 present the results of the model evaluation for Dataset 1 and Dataset 2, respectively.

In the present study, we investigated two factors that impact the ML-based model outputs in classifying benign and malicious code pieces. The first one is the method of parsing the data into sequences, which shapes the sequence characteristics in the train set and the test set. The second is the methods of disassembling the code pieces, which eventually generate data with variation in input representation. This section presents a discussion of the findings on those two dimensions.

In the past decade, malware detection has gained significant importance due to the rapid growth of information and communication technologies (ICT). The development of polymorphic and metamorphic malware has brought the need for developing novel antimalware methods since traditional detection methods have lost their effectiveness against continuously changing features of the malicious content and zero-day attacks. Deep neural networks have been proposed as a good candidate for malware detection since they provide adaptability through learning.

We investigated Android assembly code obtained through three disassembler tools in the present study: JEB, IDA, and Apktool. The assembly data obtained from those tools showed differences in their representation as input data, and those differences led to minor differences in model accuracy between JEB and IDA data. We also investigated the sequence unit in the input data (viz. instruction, basic block, method, and class). Overall, the datasets obtained by the Apktool revealed better results compared to the other two disassemblers. This performance improvement in the Apktool dataset may be explained by the presence of the path information of each function and class in the Apktool. Another likely reason is that the sequences in the Apktool dataset are considerably longer than the sequences in the other two datasets; this gave Apktool an advantage in accuracy over JEB and IDA. Future research should clarify the source of the higher accuracy in the Apktool over the others since, in their recent form, our datasets and the models do not present a clear distinction among the dataset characteristics. Also, an investigation of the overall size of the datasets on the accuracy values, besides the characteristics of the input data should be carried out. Moreover, for the RNN model implementation we employed LSTM. There is also another architecture called Gated Recurrent Unit cell (GRU). Even GRU is said to have less expressibility than LSTM in some cases, it may have some advantages such as faster training. As a future work, a comparison study between LSTM and GRU should also be carried out to see the effects on accuracy values.