On the adequacy of static analysis warnings with respect to code smell prediction

The exploited dataset reports code smell instances pertaining to 13 different types. However, not all of them are suitable for a machine learning solution. For instance, let consider the case of Class Data Should Be Private: this smell appears when a class exposes its attributes, i.e., the attributes have a public visibility. By definition, instances of this code smell can be effectively detected using simpler rule-based mechanisms, as done in the past (Moha et al. 2009).

For this reason, we first filtered out the code smell types whose definitions do not require any threshold. In addition, we filtered out method-level code smells, e.g., Long Method. The decision was driven by three main observations. In the first place, the vast majority of the previous papers on code smell prediction have used a class-level granularity (Azeem et al. 2019) and, therefore, our choice allowed for a simpler interpretation and comparison of the results. Secondly, our study focuses on the code smells perceived by developers as the most harmful (Taibi et al. 2017; Palomba et al. 2014a), which are all at class-level. Thirdly, the analyses performed in the context of our empirical study required the use of a heuristic code smell detector (i.e., Decor (Moha et al. 2009)) that has been designed and experimentally tested on class-level code smells. All these reasons led us to conclude that considering method-level code smells would not be necessarily beneficial for the paper. Nonetheless, our future research on the matter will consider the problem of assessing the role of static analysis warnings for the detection of method-level code smells.

Based on these considerations, we focused our study on the following seven code smells:

The selected code smells are those more often targeted by related research (Azeem et al. 2019). They have been also connected to an increase of change- and fault-proneness of source code (Catolino et al. 2020; Khomh et al. 2012; Palomba et al. 2018b) as well as maintenance effort (Sjøberg et al. 2012). According to previous work (Khomh et al. 2012; Palomba et al. 2018a; Yamashita and Moonen 2012), all the code smells considered let the affected source code be more prone to changes and faults in different manners. As an example, Palomba et al. (2018a) reported that the change-proneness of classes affected by the God Class smell is around 28% higher than classes not affected by the smell, while Spaghetti Code increases the change-proneness of classes of about 21%. Other empirical investigations provided different indications, e.g., Khomh et al. (2009) and Khomh et al. (2012) reported that 68% of the classes affected by a God Class are also change-prone. As a matter of fact, our current body of knowledge reports that all the code smells we considered are connected to change- and fault-proneness, but different studies provided different estimations on the extent of such connection. In addition, these code smells are highly relevant for developers that, indeed, often recognize them as harmful for the evolvability of software projects (Palomba et al. 2014a; Taibi et al. 2017; Yamashita and Moonen 2013).

In the context of our research, we selected three well-known automated static analysis tools such as Checkstyle, Findbugs, and PMD. We provide a brief description of these tools in the following:

The selection of these tools was driven by recent findings reporting that these are among the automated static analysis tools more employed in practice by developers (Lenarduzzi et al. 2020; Vassallo et al. 2018; Vassallo et al. 2019). In particular, the most recent of these papers (Vassallo et al. 2019) reported that Checkstyle, PMD, and FindBugs are actually the tools that practitioners use more when developing in Java, along with SonarQube. The selection was therefore based on these observations. In this respect, it is also worth remarking that we originally included SonarQube as well. However, we had to exclude it because it failed on all the projects considered in our study (see Section 3.1.3).

To address the research goals and assess the capabilities of the machine learning techniques for code smell detection, we needed to rely on a dataset reporting actual code smell instances. Most previous studies (Azeem et al. 2019) focused on datasets collected using automated mechanisms, e.g., executing multiple detectors at the same time to consider the instances detected by all of them as actual code smells. Nonetheless, it has been shown that the performance of machine learning-based code smell detectors might be biased by the approximations done, other than by the false positive instances detected when building the ground truth of code smells (Di Nucci et al. 2018b). In this paper, we took advantage of these latter findings and preferred to rely on a manually-labeled dataset containing actual code smell instances. Of course, this choice might have had an impact on the size of the empirical study since there exist only a few datasets of manually-labeled code smells (Azeem et al. 2019). Yet, we were still convinced to opt for this solution, as this was the most appropriate choice to do in order to have reliable results. Indeed, a dataset of real smell instances allowed us to provide reliable results on the performance capabilities of the experimented models and, at the same time, to present a representative case of a real scenario where the code smells arise in similar amounts as in our study (Palomba et al. 2018a) (Table 1).

From a technical viewpoint, the selection of projects was driven by the above requirement. We exploited a publicly available dataset of code smells developed in previous research (Palomba et al. 2015; Palomba et al. 2018a): this provides a list of 17,350 manually-verified instances of 13 code smell types pertaining to 395 releases of 30 open source systems. Given this initial dataset, we fixed two constraints that the projects to consider had to satisfy. First, the projects had to contain data for the code smells selected in our investigation (see Section 3.1.1). Secondly, we required them to be successfully built so that they could be later analyzed by the selected static analysis tools (see Section 3.1.2). These two constraints were satisfied in 25 releases of the 5 open-source projects reported in Table 2 along their main characteristics.

For the sake of completeness, it is worth reporting that most of the excluded releases/projects were due to build issues, e.g., dependency resolution problems (Tufano et al. 2017). This possibly remarks the need for additional public code smell datasets composed of projects that can be analyzed through static or dynamic tools.

This stage consisted of identifying real code smells in the considered software projects. The data collection, in this case, was inherited by the dataset exploited. While some previous studies relied on automated mechanisms for this step, e.g., by using metric-based detectors (Arcelli Fontana et al. 2016b; Khomh et al. 2009; Maiga et al. 2012), recent findings showed that such a procedure could threaten the reliability of the dependent variable and, as a consequence, of the entire machine learning model (Di Nucci et al. 2018a). Hence, in our study we preferred a different solution, namely considering manually-validated code smell instances. For all the systems considered, the publicly available dataset exploited in the empirical study report actual code smell instances (Palomba et al. 2015; Palomba et al. 2018a) and has been used in recent studies evaluating the performance of machine learning models for code smell detection (Palomba et al. 2018b; Pecorelli et al. 2019; Pecorelli et al. 2020a). For each code smell, Table 1 reports the distribution of the code smells in the dataset.

This step aimed at collecting the data of the independent variables used in our study. Each tool required a different process to collect such data:

Using these procedures, we ran the three static analysis tools on the considered software systems. At the end of the analysis, these tools extracted a total of 60,904, 4,707, and 179,020 warnings for Checkstyle, FindBugs, and PMD, respectively.

To address the first research question, we first showed boxplots depicting the distribution of the metrics and smells. Then, we computed the Mann-Whitney and Cliff’s Delta tests to verify the statistical significance of the observed differences and their effect size. With respect to other possible analyses methods (e.g., correlation), studying the distribution of warnings into the smelly and non-smelly classes not only allowed us to identify the warning types that are more related to code smells, but also to quantify the extent of the difference between the number of warnings contained in smelly and non-smelly classes.

In this RQ, we assessed the extent to which the various warning categories of the considered static analysis tools can potentially impact the performance of a machine learning-based code smell detector. To this aim, we employed an information gain measure (Quinlan 1986), and particularly the Gain Ratio Feature Evaluation technique, to establish a ranking of the features according to their importance for the predictions done by the different models. This analysis method turned to be particularly useful in our case, since it allowed us to precisely quantify the potential predictive power of each warning category for the prediction of code smells. Given a set of features F = {f₁,...,f_n} belonging to the model M, the Gain Ratio Feature Evaluation computes the difference, in terms of Shannon entropy, between the model including the feature f_i and the model that does not include f_i as independent variable. The higher the difference obtained by a feature f_i, the higher its value for the model. The outcome is represented by a ranked list, where the features providing the highest gain are put at the top. This ranking was used to address RQ₂.

Once we had investigated which warning categories relate the most to the presence of code smells, in RQ₃ we proceeded with the definition of machine learning models. Specifically, we defined a feature for each warning type raised by the tools, where each feature contained the number of violations of that type identified in a class. For instance, suppose that for a class C_i Checkstyle identifies seven violations to the warning type called “Bad Practices”: the machine learner is fed with the integer value “7” for the feature “Bad Practices” computed on the class C_i.

The dependent variable was, instead, given by the presence/absence of a certain code smell. This implied the construction of seven models for each tool, i.e., for each static analysis tool considered, we built a model that used its warnings types as features to predict the presence of God Class, Spaghetti Code, Complex Class, Inappropriate Intimacy, Lazy Class, Refused Bequest, and Middle Man. Overall, this design led to the creation of 21 models per project, i.e., one for each code smell/static analysis tool pair. For the sake of clarity, it is worth remarking that we considered each release of the projects in the dataset as an independent project. This choice was taken after an in-depth investigation of the differences among the releases available: we indeed discovered that the releases that met our filtering criteria (see Section 3.1.3) were too far in time from each other, making other strategies unfeasible/unreliable—as an example, the excessive distance among releases made not feasible a release-by-release methodology where subsequent releases are considered following a time-sensitive data analysis (Pascarella et al. 2019; Tantithamthavorn and Hassan 2018).

As for the supervised learning algorithm, the literature in the field still misses a comprehensive analysis of which algorithm works better in the context of code smell detection (Azeem et al. 2019). For this reason, we experimented with multiple classifiers such as J48, Random Forest, Naive Bayes, Support Vector Machine, and JRip. When training these algorithms, we followed the recommendations provided by previous research (Azeem et al. 2019; Tantithamthavorn and Hassan 2018) to define a pipeline dealing with some common issues in machine learning modeling. In particular, we exploited the output of the Gain Information algorithm—used in the context of RQ₂—to discard irrelevant features that could bias the interpretation of the models (Tantithamthavorn and Hassan 2018): we did that by excluding the features not providing any information gain. We also configured the hyper-parameters of the considered machine learners using the MultiSearch algorithm (Ye and Kalyanaraman 2003), which implements a multidimensional search of the hyper-parameter space to identify the best configuration of the model based on the input data. Finally, we considered the problem of data balancing: it has been recently explored in the context of code smell prediction (Pecorelli et al. 2020a) and the reported findings showed that data balancing may or may not be useful to improve the performance of a model. Hence, before deciding on whether to apply data balancing, we benchmarked (i) Class Balancer, which is an oversampling approach (ii) Resample, an undersampling method (iii) Smote, an approach including synthetic instances to oversample the minority class, and (iv) NoBalance, namely the application of no balancing methods.

After training the models, we proceeded with the evaluation of their performance. We applied a 10-fold cross-validation, as it allows to verify multiple times the performance of a machine learning model built using various training data against unseen data. With this strategy, the dataset (including the training set) was divided in 10 parts respecting the proportion between smelly and non-smelly elements. Then, we trained for ten times the models using 9/10 of the data, retaining the remaining fold for testing purpose—in this way, we allowed each fold to be the test set exactly once. For each test fold, we evaluated the models by computing a number of performance metrics, such as precision, recall, F-Measure, AUC-ROC, and Matthews Correlation Coefficient (MCC). Finally, with the aim of drawing statistically significant conclusions, we applied the post-hoc Nemenyi test (Nemenyi 1962) on the distributions of MCC values achieved by the experimented machine learners, setting the significance level to 0.05.

When addressing this research question, we were interested in understanding whether the different machine learners experimented in the context of RQ₃ were able to detect code smell instances that are not detected also by other techniques. If this was the case, then it meant that different automated static analysis tools would have had the potential to predict the smelliness of classes differently, hence possibly enabling the definition of a combined machine learning mechanism that it could have further improved the code smell detection capabilities. In other terms, the analysis aimed at understanding how many true positives can be identified by a specific model alone and how many true positives can be correctly identified by multiple models. To this purpose, for each code smell type, we compared the sets of correctly detected instances by a technique m_i with those identified by an alternative technique m_j using the following overlap metrics (Oliveto et al. 2010): where \(correct_{m_{i}}\) represents the set of correct code smells detected by the approach m_i, \(correct_{m_{i} \cap m_{j}}\) measures the overlap between the set of true code smells detected by both approaches m_i and m_j, and \(correct_{m_{i} \setminus m_{j}}\) appraises the true smells detected by m_i only and missed by m_j. The latter metric provides an indication of how a code smell detection technique contributes to enriching the set of correct code smells identified by another approach.

We also considered an additional orthogonality metric, which computed the percentage of code smell instances correctly identified only by the prediction model m_i. In this way, we could measure the extent to which the warning types of a specific static analysis tool contributed to the identification of all correct instances identified. Specifically, we computed:

While different models can identify different correct code smell instances, they can also identify different false positives. This means that the complementarity of the models does not necessarily mean that their combination would result in a better model. In the next Section we show how to build a combined model and compare it with the individual ones.

In this research question, we took into account the possibility to devise a combined model that mixes together the outputs of different static analysis tools.

Starting from all warning types of the various tools, we have proceeded as follows. In the first place, we built a new dataset where, for all classes of the systems considered, we reported all the warnings raised by all tools. This step led to the creation of unique dataset that combined all the information mined in the context of our previous research questions. In the second place, we have re-run the Gain Ratio Feature Evaluation (Quinlan 1986) in order to globally rank the features and discard those that, in such a new combined dataset, did not provide any information gain.

After discarding the irrelevant features, we have followed the same steps as RQ₃ with the aim of conducting a fair comparison of the combined model with the individual ones previously experimented. As such, we trained the model using multiple classifiers appropriately configured using the MultiSearch algorithm (Ye and Kalyanaraman 2003) and considering the problem of data balancing (Pecorelli et al. 2020a). Afterwards, to verify the performance of the combined model, we adopted the same validation strategy as RQ₃ and compared it with the values of F-Measure, AUC-ROC, and Matthews Correlation Coefficient obtained by the individual models. Finally, we used the Nemenyi test (Nemenyi 1962) for statistical significance.

To address RQ₆, we had to first select an existing solution to compare with. Most of the previous studies (Al-Shaaby et al. 2020; Azeem et al. 2019; Kaur et al. 2021) experimented with various machine learning techniques, yet they all employed code metrics as predictors. As an example, Maiga et al. (2012b) characterized God Class instances by means of Object-Oriented metrics. Similarly, other researchers have attempted to verify how different machine learning algorithms work in the task of code smell classification without focusing on the specific features to use for this purpose (Azeem et al. 2019). Hence, we decided to devise a baseline machine learning technique that uses code metrics as predictors. In this respect, we computed the entire set of metrics proposed by Chidamber and Kemerer’s suite (Chidamber and Kemerer 1994) with our own tool and use them as features.

After computing the code metrics, we followed exactly the same methodological procedure used in the context of RQ₃ and RQ₅. As such, the baseline machine learner aimed at predicting the presence/absence of code smells. Also in this case, we experimented with various machine learning algorithms, finding Random Forest to be the best one. When training the baseline, we took care of possible multi-collinearity by excluding the code metrics providing no information gain, other than tuning the hyper-parameters by means of the MultiSearch algorithm (Ye and Kalyanaraman 2003). In terms of data balancing, we verified what was the best possible configuration, benchmarking Class Balancer, Resample, Smote, and NoBalance: Smote was found to be the best option.

We applied a 10-fold cross validation on the dataset, so that we could have a fair comparison with the approach devised in RQ₅—note that we did not consider a full comparison with the individual models experimented in RQ₃ since these were shown already to be less performing. The accuracy of the baseline was assessed through F-Measure, AUC-ROC, and MCC. Finally, we executed the post-hoc Nemenyi test (Nemenyi 1962) on the distributions of MCC values achieved by the baseline and the combined machine learner output by RQ₅, setting the significance level to 0.05.

In this research question we performed a complementarity analysis between the warning- and the metric-based Prediction Models. In order to perform such a complementarity analysis, we followed the same methodology applyed for RQ₄. In particular, for each actual smelly instance, we computed the overlap metrics described in Section 3.3.4, i.e., \(correct_{m_{i} \cap m_{j}}\) and \(correct_{m_{i} \setminus m_{j}}\).

To study the performance of a machine learner that exploits both static analysis warnings and code metrics, we have proceeded in a similar manner as the other research questions, After combining all the metrics experimented so far in a unique dataset, we re-run the Gain Ratio Feature Evaluation (Quinlan 1986) to understand the contribution provided by each of those metrics. As previously done, we discarded the ones whose contribution was null. Afterwards, we followed the same steps as RQ₅ and compared the performance of the combined model to the previously built models using F-Measure, AUC-ROC, and MCC, other than the Nemenyi test (Nemenyi 1962) for statistical significance.