BCGen: a comment generation method for bytecode

Java bytecode is compiled intermediate code that needs to be converted to machine code by the Java Virtual Machine (Lindholm et al. 2013). Java bytecode instructions contained in the Java class files enrich semantic information and direct the execution of the source code. The bytecode is usually executed directly on the virtual machine, which further compiles the bytecode into machine code to improve performance.

To run a Java program, the source code must be converted into a CLASS file with a suffix of .class, that is, a binary bytecode file is interpreted and executed by the virtual machine after being compiled by the compiler. We can compile the source code into a CLASS file through the IDE tool or the command line, so that we can get the bytecode corresponding to the source code. Figure 1 shows an example of the source code of a Java method and its bytecode. The upper left of Fig. 1 is a Java source code. The Java source code here is a |getString()| method, whose main function is to return the given string. The upper right part is the hexadecimal form of bytecode generated after compilation. We can view this hexadecimal format corresponding to the bytecode with a hexadecimal viewer. Since this bytecode contains a lot of other information, we highlight (the blue highlighted area) the hexadecimal format corresponding to the Code area of the |getString()| method bytecode. The part outlined in the red box corresponds to the hexadecimal code of the opcode sequence in the Code area.

The bottom of Fig. 1 is the result of the disassembly of the bytecode. We can use the “|Javap|” (an anti-parsing tool that comes with |JDK|) command to disassemble the bytecode to get this part of the content. We can see that the core part of the bytecode corresponding to the Java method is the Code area, which contains the bytecode instruction, the line number table (LineNumberTable), and the local variable table (LocalVariableTable). The LineNumberTable is used to describe the correspondence between Java source line numbers and bytecode line numbers, and the LocalVariableTable describes the local variables of the Java method.

In bytecode instructions, one byte represents one instruction. For example, the bytecode in the hexadecimal form corresponding to the opcode “|aload_1|” is “0x2B”. An instruction may or may not have parameters. If there are parameters, the following bytecode is its parameter. If there is no parameter, the following bytecode is the next instruction. As shown in Fig. 1, |invokevirtual| means: use the object pointed to by the data of the reference type at the top of the operand stack as the method receiver, and then call the instance constructor method, the private method or the method of its parent class of this object. The following “|#3|” is the constant pointed to by this instruction, “|//|” represents a specific form of this constant. It shows that bytecode instructions mainly contain information about program execution. Therefore, these instructions provide us with rich semantic information for understanding the bytecode.

The first part of Fig. 2 shows the process of collecting the dataset. The original source of the dataset comes from the most popular projects in the Maven repository. We first manually download the jar package of the Java project on the Maven repository according to the popularity ranking (the usages of the jar packages). We finally collected 665 jar packages. Since the directly downloaded projects may not meet our needs, it is necessary to further process and analyze the downloaded projects. Our goal is to generate the corresponding comment for method-level bytecode. However, the bytecode itself does not contain comments, so we need to find the source code corresponding to the bytecode to extract the comments. Specifically, we use the method of regular expression matching to find all the methods in the Java class and get the comment located before the method header. However, many comments are too long to be used directly as our data. The total number of words in the comment is very large. Take the |getImage()| method as an example, its comment is shown in Fig. 3. This example shows that the comment content includes the description of method functions, parameters, return values, and the detailed introduction of methods. The number of words in the comment is very large. If this comment is used directly, it is not conducive to model learning. The first sentences of Java methods comment usually describe the functionalities of a Java method according to Javadoc guidance.⁴ Following the way of Hu et al. (2020), we also take the first sentence of the comment as the target comment.

In order to reduce the vocabulary size, reduce the dimension of the language model, and obtain more accurate text analysis and expression, we do further processing on the extracted comments. Specifically, we perform stem extraction and part-of-speech restoration on the comment text, and the specific processing is as follows.After extracting the comment, we can locate the corresponding bytecode fragment through the method name. Finally, there are 498 jar packages meet our data requirements, i.e., these projects contain both bytecode and the corresponding source files (with comments).

In addition, we also performed template deduplication on the collected dataset. Assuming that \(<>\) represents any character, then the sentences “get the type of error” and “get the type of event” can be represented by the template “get the type of \(<>\)”. If the dataset contains too many such similar sentences, it is difficult for us to judge whether the high-scoring (e.g., BLEU) model has really learned the knowledge or just correctly guessed the repeated words in the template. Therefore, We need to perform template deduplication. Specifically, we use the AEL tool (Jiang et al. 2008), which is one of the state-of-the-art log parsing approaches for template detection. Its template detection accuracy can reach 90%. The AEL tool is used to perform template detection on the bytecode-comment pairs (A bytecode-comment pair is called an instance) we obtained. For instances in all templates, we keep only two of them that contain more than 80% of the same token. This ensures that for the entire dataset, at most only two instances are very similar. Since template detection does not change the token of the original dataset and the model structure, its performance affects the effect of deduplication according to the template, which ultimately affects the size of the deduplicated data set. At last, we collect 55,130 bytecode-comment pairs.

Take the |peek()| method as an example, which is shown in Fig. 4. Its comment is “return the first value of this queue, null if empty”. It can be found that the bytecode is composed of an opcode sequence. Compared with the source code, it is difficult to understand the information described by bytecode only.

Bytecode files are unreadable binary files. In order to generate code comments, we need to extract key information from them. By compiling Java source code, we can get the Java bytecode. As described in Sect. 2, the core part of the bytecode corresponding to the Java method is the Code area, which contains the bytecode instruction, the line number table (LineNumberTable, in Fig. 1), and the local variable table (LocalVariableTable, in Fig. 1). The bytecode instructions consist of operators and operands, which contain rich semantic information about the program. Therefore, we mainly extract the semantic information and the CFG in the Code area to express the characteristics of the bytecode.

For the semantic information extraction, we mainly focus on the tokens in the Code area. After being decompiled, the Code area may contain a large number of tokens. Using them directly will make the vocabulary too large, and many unknown words may appear in the prediction phase of the model, which has an impact on the prediction accuracy of the model. However, most of these tokens meet the camel naming rules or connect by “.”, so we split the tokens by camel word segmentation and “.” to reduce the vocabulary, so as to alleviate the problem of Out-of-vocabulary.

We deal with the tokens as follows:The final token sequence of the |peek()| method is shown below:

<BEG> public final simplevar peek code aload_0 invokevirtual #19 method peek node lakka dispatch abstract bound node queue node astore_1 aload_1 ifnull<NUM> aload_1 getfield #20 field akka dispatch abstract bound node queue node value ljava lang object goto <NUM> aconst_null areturn <END>

Compared with source code, the gap between bytecode and comment is larger, so the translation between bytecode and comment is challenging. An easy way to model bytecode is to treat it as plain text. However, this ignores structural information and may reduce the accuracy of the generated comments. In order to learn the structural information of the bytecode, we introduce the CFG into the model.

A CFG is a graphical representation of the control flow during the execution of a program. It can represent the possible flow of execution of all basic blocks during code execution, and it can also reflect the real-time execution of a piece of code. It contains code structural information, which is helpful for the model to understand bytecode and generate comments. However, we cannot simply feed the CFG into BCGen, which is a sequence processing model. At present, there are related methods to serialize the CFG and capture the features of the CFG, such as the studies (Ding et al. 2014; Phu et al. 2019; Ramu et al. 2020). Ding et al. (2014) constructed an executable tree by removing the reverse edges of the CFG, and then connect the paths from all root nodes to leaf nodes of the executable tree as a sequence representation of the graph. Finally, n-grams are used to extract sequence features. Due to the intersection of the paths from the root node to the leaf node, it may lead to a large number of token repetitions, and the final sequence representation is too long. The study Phu et al. (2019) is an improved version of Ding et al. (2014) that reduces the time complexity of feature extraction. If we directly apply these methods to serialize the CFG, the generated sequence will be too redundant and long, and the model training effect is poor and memory-consuming. Ramu et al. (2020) proposed a customized control flow analysis method (BCFA) for the efficiency of implementing source code analysis using large-scale control flow graphs (CFGs). However, this method needs to perform static analysis on the CFG before selecting the traversal strategy, and construct a decision tree to select the optimal traversal strategy. On the one hand, for our dataset with a small amount of data, the time saved by traversal optimization may not be worth the time cost of strategy selection. On the other hand, different traversal strategies will lead to inconsistent representation of the final generated CFG sequences, which may have an impact on the performance of the model. Therefore, a simple traversal method is used to construct the sequences. Different from the above methods, We propose to traverse the CFG using a preorder traversal method and convert it into a sequence. The details are presented in Algorithm 1. The input of Algorithm 1 is the root node of the CFG and a list used to mark the visited nodes. This list can prevent the program from infinite loops due to the ring structure of the graph. The algorithm is implemented recursively, traversing each CFG node in order and recording it in the defined string, and finally, we can get the converted string sequence.

We generate a control flow graph in the form of a DOT file through the bytecode. The DOT file is a text file that describes the constituent elements of a diagram and the relationships between them. In addition, the DOT file can be visualized with the |GraphVize| tool.⁶ The principle of generating the CFG from the bytecode file is to decompile the bytecode into a three-address code, and identify some basic blocks. Blocks are connected to form a control flow graph, which can be described by the DOT language. In our work, we use the soot tool (Lam et al. 2011) to generate DOT files from bytecode files. Soot is a software engineering tool for analyzing and optimizing Java programs. It is often used for static or dynamic analysis of code or logs. The input of the soot tool is a bytecode file. Using its CFGViewer method, the bytecode can be parsed into a three-address code and identify basic blocks without additional parameter settings, and the final generated CFG is saved in a DOT file. In the process of dataset collection, using Soot has an 80% success rate to generate CFG. We use Soot to convert the CLASS file into DOT file and the data contained in the DOT file is the CFG. Then we serialize the CFG in the DOT file to get the text that can be input to the model. Figure 4 shows the CFG corresponding to the |peek()| method.

Finally, we deal with the sequence obtained by the CFG in the way of preorder traversal as follows:The final CFG sequence of the |peek()| method is shown below:

<BEG> r0 = this r3 = akka dispatch abstract bound node queue r0 r1 = r3 peek node if r1 == null goto label1 r2 = r1 value goto label2 label return r2 if r1 == null goto label1 label r2 = null <END>

In this part, we will introduce the details of BCGen. Figure 5 illustrates the model framework of BCGen. It can be observed that BCGen is mainly composed of three parts, two encoders (called |Token Encoder| and |CFG Encoder| respectively) and one decoder. Since semantic and structural information are different modalities of bytecode, we need to feed the two sequences (token sequence of bytecode and CFG sequence) to the model separately. BCGen uses two encoders to encode token and CFG sequences, respectively, so that one encoder of BCGen can learn the semantic information of the bytecode from the token sequence, and the other encoder can learn the structural information from the CFG sequence. The ability of the encoder and decoder to extract features from input sequences determines the performance of the model. Common sequence feature extraction models include RNN, LSTM, GRU, Transformer (Zhou et al. 2021; Wang et al. 2019) and so on. One way to improve accuracy is to strengthen the neural architecture. In the above models, the Transformer can better handle long-term dependencies, while GRU has fewer model parameters. In the NLP community, Transformer has achieved the most advanced results (Devlin et al. 2018; Dong et al. 2019; Radford et al. 2019), outperforming RNN, for a variety of NLP tasks. Therefore we adopt the Transformer-based encoder–decoder structure as the model backbone. However, the model requires two encoders to encode the input sequence, and our early attempts showed that using the Transformers for both encoders would result in a model with too many parameters and poor accuracy. Therefore, we replace one of the encoders with GRU to reduce model parameters and guarantee model accuracy. Next, we will introduce the details of these three parts separately.

We used the basic Seq2Seq model(Sutskever et al. 2014), the attention-based Seq2Seq model(Luong et al. 2015) and the Hybrid-deepcom model (Hu et al. 2020) as the baselines. The Seq2Seq model is often used in machine translation, text summarization and other tasks. In general, the Seq2Seq model can achieve better results after introducing attention mechanism. Hybrid-deepcom is one of the state-of-the-art models for code comment generation. In essence, it is also a Seq2Seq model, which uses GRU as encoder and decoder, and contains two encoders to extract the syntax and semantic information of the source code respectively.

The hardware platform of our experiment adopts NVIDIA RTX 3090, Intel i9 10980XE CPU @ 3GHz with 256GB memory. The training parameters are set as follows: the training batch size is 32, the learning rate is 0.001, the maximum number of iterations is 10. We use twelve Transformer blocks with eight heads in each block, and the embedding dimension is set to 768.

The length of the input sequence is an important parameter of the model. In order to set the input sequence length reasonably, we counted the length distribution of token sequence, CFG and the comment of each method, as shown in Fig. 6. By calculation, when the maximum length of token and CFG sequences is set to 384, and the maximum length of the comment is set to 64, more than 90% of the methods can be covered. Therefore, we finally set the token and CFG sequences’ maximum length to 384, and the comment maximum length to 64.

In this paper, we use BLEU(Papineni et al. 2002), METEOR(Banerjee and Lavie 2005), ROUGE-L(Lin 2004) to measure the effect of comments generated by the model.

BLEU is an indicator commonly used in the NLP field to measure the similarity of two texts. We use BLEU to calculate the similarity between the comments generated by the model and the reference comments, so as to measure the comments generation effect of the model. The BLEU score ranges from 1 to 100 as a percentage value. The higher the score, the higher the similarity between the two comments. It is calculated as follows:where \(p_{n}\) is the ratio of subsequences of length n in the candidate sequence that are also in the reference sequence. N is the maximum number of grams. And BP is brevity penalty:where \(l_{c}\) is the length of the candidate translation and \(l_{s}\) is the length of the effective reference sequence. To better illustrate the results, we evaluate the generated comments by different smoothing methods. Chen and Cherry (Chen and Cherry 2014) introduced 7 smoothing techniques that work better for sentence-level evaluation. In this study, we use smoothing functions 3.

METEOR calculates the similarity scores between a pair of the generated sequence and reference sequence by:Where Pen is the penalty term, which is calculated according to the number of chunks (ch) and the number of matches (m):and \(F_{mean}\) is the harmonic mean of precision(P) and recall(R).The \(\alpha\), \(\beta\) and \(\gamma\) above are three penalty parameters whose default values are 0.9, 3.0 and 0.5, respectively.

ROUGE-L computes F-score based on the length of the longest common subsequence (LCS). Suppose the lengths of the generated comment X and reference comment Y are m and n, it is computed as:The parameter \(\beta =P_{lcs}/R_{lcs}\) and \(F_{lcs}\) is the value of ROUGE-L. One advantage of using LCS is that it does not require consecutive matches but in-sequence matches that reflect sentence-level word order as n-grams.

In our experiments, we mainly focus on the following research questions:We ask RQ1 to evaluate the BCGen model compared with other deep learning models. We ask RQ2 in order to evaluate the impacts of the CFG on the accuracy of the comment generation. For RQ3, we want to examine the effect of the bytecode length on the quality of comments generated by BCGen.

Different from the current code comment generation, our work directly generates comments for the bytecode, that is, directly uses the bytecode as the model input, and outputs the corresponding natural language comments. Since there is little relevant bytecode comments generation research, we mainly discuss the approaches of comment generation at the source code level.

The general method of code comment generation is shown in Fig. 9. First, the code is downloaded from the open-source website and processed to form a dataset, and then the comments are generated by the methods such as heuristic templates(Haiduc et al. 2010a; Rodeghero et al. 2015; McBurney and McMillan 2015), information retrieval(Wong et al. 2013; Rodeghero et al. 2017) and deep learning(Alon et al. 2018; Hu et al. 2018a).

Traditional automatic code comment generation methods (Wong et al. 2013; Haiduc et al. 2010b; Moreno et al. 2013; Rodeghero et al. 2017) are mainly based on heuristic templates and information retrieval techniques. They directly extract useful information from source code to generate comments. Most of these methods have limited generalization ability. In recent years, deep learning techniques have developed rapidly, and many studies have been devoted to generating code comments using deep learning methods (Alon et al. 2018; Haque et al. 2020; Hu et al. 2018a; Iyer et al. 2016). What’s more, deep learning methods have also promoted the development of traditional retrieval-based comment generation methods (Li et al. 2021; Zhang et al. 2020; Wei et al. 2020). Powerful deep learning models can extract more abstract information from the source code, and further improve comment generation accuracy. Since we also employ the deep learning model to generate bytecode comments, we will discuss the approaches based on deep learning models.

Iyer et al. (2016) proposed an RNN network with attention to generate comments describing |C#| code snippets and |SQL| queries. The idea is to treat the source code as plain text, and then model the conditional distribution of comments through an RNN network. This work is the first one to use an encoder–decoder framework for code comment generation.

Many researchers believe that using source code information alone is not enough to generate high-quality comments, so they use ASTs that contain structural and semantic information to generate code comments. Hu et al. (2018a) found a problem that AST is a tree and neural machine translation model relying on sequence input. So they proposed to use a traversal method called structure-based traversal (SBT) to convert an abstract syntax tree (AST) into a traversal sequence, and then used a Seq2Seq attention model to convert the code-abstracted AST sequence into a digest. Different from Hu et al. (2018a), Shido et al. (2019) directly proposed a Tree-based LSTM model, which starts from the leaf node and encodes from the bottom up. Furthermore, Alon et al. (2018) randomly selected leaf node pairs on the abstract syntax tree to generate multiple paths, and then encoded each path using the RNN model to obtain the information of these paths.

The above methods are all based on one kind of information to generate comments. In recent years, many models fuse multiple kinds of information as input. Hu et al. (2020) proposed a Hybrid-deepcom model. This model has two encoders, and utilizes both the lexical information of the source code and the grammatical information of the AST. LeClair et al. (2019) proposed a neural model that combines the AST source code structure and words in the code to generate coherent comments of Java methods. The specific approach is to design a model architecture to process word sequences and SBT/AST sequences in different recurrent networks with attention mechanisms. Haque et al. (2020) argued that using only the internal information of the code fragment would limit the performance of the model. Therefore, they proposed a method to use file context information to help generate code comments. What’s more, Hu et al. (2018b) combined the API document information to generate code comments on the basis of the source code.

In addition to the above methods, there are many other methods of code comment generation. Allamanis et al. proposed to model source code using graph-based neural networks (Allamanis et al. 2017) as well as convolutional networks with attention (Allamanis et al. 2016). Wang et al. (Wang et al. 2020) presented a code comment generation approach using the hierarchical attention network by incorporating multiple code features.

Currently, bytecode has been used for many software tasks, such as binary code search (Yang et al. 2021; Xue et al. 2018; Tian et al. 2021), malware detection (Daoudi et al. 2021; Rozi et al. 2020; Zhang et al. 2021), vulnerability detection (Guo et al. 2019; Tian et al. 2020), code clone detection (Yu et al. 2017; Keivanloo et al. 2012), code similarity detection Liu (2021). Yang et al. (2021) proposed the Codee model. In terms of processing binary code, they proposed an optimization function of basic block embedding generation in binary codes as a network representation learning problem to capture the semantic information of basic blocks and the structural information of CFG. BinDeep proposed by et al. Tian et al. (2021) obtained the instruction sequence of each function in the binary code by decomposing the instructions, and then used the instruction embedding model to vectorize the extracted instructions. Rozi et al. (2020) proposed a method to detect malicious JavaScript using deep neural networks to extract bytecode sequence information. It used a compiler to generate a sequence of bytecode that corresponds to an abstract form of machine code. Guo et al. (2019) constructed a software vulnerability detection system based on deep learning and bytecode, in which bytecode slices were converted into numeric vectors as the input of neural networks. Yu et al. (2017) proposed a novel code clone detection method based on Java bytecode. This method can simultaneously detect code clones at both the method level and block level. In order to improve accuracy, the similarities of both methods are call sequences and instruction sequences during the process of code clone detection. A common process of the above work is to convert the bytecode into vector sequences to extract features, and then apply them to different software tasks, while none of these work focuses on the comment generation at bytecode level.