Rubbing salt in the wound? A large-scale investigation into the effects of refactoring on security

The empirical study is based on three levels of analysis. Following the preliminary investigation by Abid et al. (2020)—who observed a correlation between the amount of refactoring operations applied and security metrics—we aimed at assessing the security implications of refactoring operations on security indicators in an effort to provide insights into the potential compromise a developer should pay attention to while improving source code quality.

We start facing this research objective using two analyses: the first focused on security-related metrics that indicate portions of source code whose characteristics may lead the code to be more exposed to security risks (Alshammari et al. 2009); the second targeting technical debt (Curtis et al. 2012) that highlights the design and implementation issues that might represent exploitable security flaws. These two analyses are by nature complementary: security-related metrics focus on weak constructs implemented in the source code, while security technical debt measures on higher-level poor design or implementation solutions that might possibly impact the security profile of an application. As further explained in Section 2.3, we conducted these analyses by measuring developers’ activities, and their implications for source code security by running tools and analyses on commits where refactoring has been applied. These goals led to the formulation of the following two research questions:

While the first two research questions allowed us to uncover possible negative effects given the application of refactoring operations on security, these were not sufficient nor comprehensive. Both security-related metrics and technical debt focus on potential risks for source code security; yet, this does not still clarify if and how refactoring has an impact on the introduction of real security threats. For this reason, we continued our empirical investigation by assessing how the application of refactoring operations over the change history of software projects leads to the introduction of known software vulnerabilities. This reasoning led to our last research question:

The study can be configured as a quantitative investigation (Sukamolson 2007) where we seek to find statistically significant findings from a large amount of data. While the next sections detail the data collection and analysis procedures used to address our research questions, Fig. 2 overviews the methodology employed in our study. In short, when addressing RQ₁ and RQ₂ we first run three tools, namely an automated refactoring detector called RefactoringMiner (Tsantalis et al. 2018), a security metric tool named Surface, and a static code analyzer called SonarQube over all the commits of the considered projects. Afterward, we use the data collected to compute the difference in terms of security metrics and debt between the refactoring commits and their predecessors to indicate how the refactoring operations have changed these measures. Finally, the variation of security metrics and debt were used as dependent variables of Multinomial Log-Linear regression models (Theil 1969) that allowed us to identify which refactoring operations are statistically related to their increase or decrease while controlling for factors like complexity, lines of code, and code churn.

As for RQ₃, we mined the vulnerability-fixing commits of known vulnerabilities affecting the software projects considered in our study and available on a public dataset of vulnerabilities, namely, the National Vulnerability Database (NVD) (2023b). Then, we employed an automated mechanism based on the SZZ algorithm (Sliwerski et al. 2005) to identify the commits responsible for the introduction of those known vulnerabilities and combined this information with the one from RefactoringMiner to obtain the number of times refactoring operations are likely to have contributed to a known vulnerability. A manual qualitative investigation later contextualized the statistical analyses to understand further and discuss the quantitative results. The detailed methodological steps adopted to collect the described data are reported in Section 2.3.

The context of the study was composed of open-source software projects and, in particular, their change history information. In this respect, we exploited the Technical Debt Dataset (Lenarduzzi et al. 2019), which is a curated collection of data coming from 39 Java projects mainly from the Apache Software Foundation ecosystem. Despite belonging to a single ecosystem, the majority of such projects were originally selected by following the diversity guidelines introduced by Nagappan et al. (2013), i.e., they were selected by addressing the representativeness of projects in terms of age, size, and domain, and the Patton’s “criterion sampling” (Patton 2002), namely, they are more than four years old, have more than 200 commits and 100 classes, and have more than 100 issues reported in their issue tracking system. As such, this dataset minimizes by design possible threats to external validity. To further verify the properties of this dataset, we have manually investigated the corresponding Github repositories and discovered that all of them adhere to a strict code of conduct (Tourani et al. 2017) and regularly review source code to improve their quality processes (Pascarella et al. 2018). This analysis further confirmed the suitability of the dataset. It is important to note that nine of the considered systems also appear in the National Vulnerability Database (NVD) (2023b), which was initially developed by the U.S. NIST Computer Security Division (2023a) to collect and provide public information about known vulnerabilities affecting software systems and their causes. Such a database includes a comprehensive set of publicly known vulnerabilities: each of them is described through CVE (Common Vulnerabilities and Exposure 2023a) records and is enriched with additional pieces of information such as external references, severity (computed using the Common Vulnerability Scoring System - CVSS), the related weakness type (Common Weak Enumeration - CWE), and the known affected software configurations (Common Platform Enumerations - CPEs). NVD aggregates information from multiple data sources and is widely considered a reliable data source (Alhazmi et al. 2007; Huang et al. 2010; Zhang et al. 2011).

While all the projects were considered when addressing RQ₁ and RQ₂, only the nine systems overlapping with the NVD could be used for RQ₃ as these are the only ones for which we could obtain data on the known vulnerabilities affecting them. Table 1 reports the main characteristics of the projects in our context—we report statistics on their change history with a particular focus on the refactoring operations observed.

This section describes how we collected each piece of information to address our research questions: developers’ refactoring operations, security metrics, security-related technical debt, and known vulnerabilities that affected the projects considered.

After collecting the data required to address our research questions, we proceeded with the statistical modeling and the subsequent interpretation.

Once we completed the statistical modeling, we proceeded with the verification of the high-level hypotheses formulated in Table 2. More specifically, we first refined them to derive more concrete null and alternative hypotheses to test the research questions in this study. In the cases of Move Package, Extract Interface, Move Class, Move Method, and Move Attribute refactoring, we defined the following null hypothesis:

Our alternative hypothesis was, instead:

In the context of RQ₁ and RQ₂, we rejected the null hypotheses if the coefficients of the statistical models built to understand the increase and decrease of the security properties were not significant or negative. In the latter case, the coefficients of the Multinomial Log-Linear model would indicate that refactoring operations tend to increase the likelihood of security metrics/debt being stable, hence rejecting the null hypothesis in favor of the alternative one. As for RQ₃, we run the non-parametric Mann-Whitney U test (Mann and Whitney 1947) (with α =0.5) on the distribution of refactoring operations within VCCs and non-VCCs commits. We rejected the null hypothesis if α > 0.05. We also measured the effect size of the differences identified in the two distributions using Cohen’s d (Cohen 2013). We followed well-established thresholds for interpretation: 0.2 for Small, 0.5 for Medium and 0.8 for Large effect size (Cohen 2013).

With respect to Extract Superclass, Extract Method, Extract & Move Method, Pull Up Method, and Pull Up Attribute refactoring, our null hypothesis was:

The alternative hypothesis in this case was:

In RQ₁ and RQ₂, we rejected the null hypothesis if we observed (i) both significant coefficients and (ii) positive coefficients in the statistical model built to understand the increase of security metric/debt values and negative coefficients in the statistical model built to understand the decrease of security metric/debt values. In the latter case, indeed, the statistical model coefficients would tell us that the application of refactoring operations tends to increase the likelihood of the metrics/debt being increased, hence indicating their deterioration. As for RQ₃, we still relied on the same outcomes and interpretation of the Mann-Whitney U test (Mann and Whitney 1947) and Cohen’s d (Cohen 2013).

Finally, when it comes to Inline Method, Move & Inline Method, Push Down Method, and Push Down Attribute refactoring, the null hypothesis was set to:

The alternative hypothesis in this case was:

As for RQ₁ and RQ₂, we rejected the null hypothesis if we observed (i) both significant coefficients and (ii) positive coefficients in the statistical model built to understand the decrease of security metric/debt values and negative coefficients in the statistical model built to understand the increase of security metric/debt values. In the latter case, the statistical coefficients would indicate that the application of refactoring operations has the tendency to increase the likelihood of the metrics/debt being decreased, which would mean that the security profile would improve. In RQ₃, we relied on the outcomes and interpretation of the Mann-Whitney U test (Mann and Whitney 1947) and Cohen’s d (Cohen 2013).

In order to allow our study to be verified and replicated, we have published the complete raw data, along with the data collection and analysis scripts in our online appendix (Iannone et al. 2022). The ‘README.md' file contains more precise instructions on how to use our artifacts to replicate the study.

In RQ₁, we sought to understand the relation between refactoring and security metrics. Table 6 reports for each refactoring type the sign of the logit coefficients (within a circle) and the value of the ORs obtained for the Multinomial Log-Linear models built to understand the decrease and increase of the security metrics considered in the study. The coefficients of the variables that turned out to be statistically significant are reported with a colored symbol (green for , red for ), otherwise are left white. In addition, the cells with a gray shade indicate that the impact of that type of refactoring rejects the null hypothesis formulated in Section 2.5—i.e., the impact turned out to be in line with our expectations. For the sake of readability, we did not report the coefficient signs and ORs values of the confounding factors (LOC, WMC, and code churn) considered when building the models, but we discuss their role in our analysis and report the full results in our online appendix (Iannone et al. 2022). Looking at the table, we can immediately observe that the refactoring types Move Method, Move Attribute, Extract Superclass, and, to a lesser extent, Push Down Method, always had positive coefficients in all the 13 models built for the 13 security metrics; this means that moving code components (i.e., attributes or methods) from a class to another or optimizing the degree of code reuse (i.e., creating better class hierarchies) have the effect of varying the security profile of an application, either positively or negatively. Moreover, the Extract Superclass refactoring had a particularly strong impact on CCR and CSCR metric, i.e., their ORs are the largest among all the other models. As such, extracting new classes in hierarchies impacts (i) the ratio of the number of critical classes to the number of classes in the entire project (ii) and the number of superclasses in all the class hierarchies (CSCR), respectively. Although these findings suggest that these refactoring types have a random effect on security metrics, it is reasonable to believe that the impact is determined by their specific application, i.e., the security profile is affected differently depending on how developers apply the refactoring operations.

Surprisingly, refactoring types for which we expected no impact, such as Extract Interface and Move Package, still exhibited a statistically significant mixed effect on the security metrics. Between the two, Extract Interface showed a clearer behavior. On the one hand, it tends to increase the value of CC, CCE, CCR, and CMR metrics—i.e., increasing the number of critical classes. On the other hand, it keeps reducing CCVA, CIVA, and CM metrics—i.e., reducing both the accessibility of instance and class variables, and the number of classified methods. This means that re-organizing the classes’ interfaces helps keep the number of critical attributes and methods under control, still increasing the risk of introducing too many critical classes. Differently, only for the CIVA metric, the Move Package refactoring matched our expectations: moving classes among packages does not affect this metric at all. This might be explained by the fact that commits applying package restructuring are generally not done in a fully-isolated manner but are applied in the context of other changes—which have a mixed impact.

Extract Method, Inline Method, and Move Class still exhibited mixed effects on the various security metrics, but with much lesser significance than other refactoring types. The only cases where the null hypotheses were rejected were for Extract Method for CM and CME metrics, and also for Inline Method for CM metric. This is quite straightforward to comprehend. Extract Method creates new methods from a piece of code in existing methods, likely introducing new classified methods if the extracted logic deals with classified attributes. At the same time, Inline Method refactoring eliminates redundant methods, with a high chance of removing classified methods and reducing access to classified attributes. This was the case of project Conversation at the revision 14cfb609. In such a commit, the utility class CryptoHelper was streamlined into three new classes, all placed in package crypto/sasl. Additionally, the XmppConnection class—in change or managing XMPP connections—was refactored to inline the two methods sendSaslAuthPlain() and sendSaslAuthDigestMd5() into processStreamFeatures(). Indeed, the two removed methods were only called by processStreamFeatures(), so driving the developer to apply two Inline Method refactorings—the commit message confirms this intentions, i.e., ‘Refactor authentication code’. In this respect, sendSaslAuthPlain() was a classified method, as it access to security-sensitive data, i.e., account instance variable in this case. Hence, its removal led to the reduction of one from the CM metric. This variation translates into a reduction of the overall application’s attack surface. Indeed, the metrics proposed by Alshammari et al. (2012) penalize those classes exposing too many methods that access classified components, as attackers might leverage them to carry out attacks. This explains why Inline Method refactorings have been seen as beneficial from this perspective is beneficial. Such an example also opens an interesting observation. While a securely-designed class should minimize the number of methods with the responsibility of accessing security-critical data, good design practices for maintainable code recommend creating many small and cohesive methods. Thus, creating both maintainable and secure code demands particular care not to create too many classified methods but still avoiding making poorly cohesive and long methods. In other words, accessing security-critical data should be reserved for only an essential set of elected classes and methods. Similarly, in commit 43531113 of the same project, an Extract Method refactoring was applied on sendBindRequest() to extract sendIqPacket() method, which happened to access security-sensitive data—so, it was branded as a new classified method. While this method is not necessarily a security issue, its presence increases the application’s attack surface, giving attackers an additional method to leverage for its purpose.

All the other refactoring types, i.e., Move Class, Pull Up Attribute, Push Down Attribute, and Pull Up Method, appears to have limited or no impact on any security metrics. Perhaps more interestingly, the composite refactoring types considered in the study, namely Extract & Move Method and Move & Inline Method, seem to show a “mixture” of the behaviors of their individual operations. This had the curious effect of having almost no statistical significance for all 13 models. Yet, this could also be caused by the limited amount of composite refactorings observed in our dataset.

When considering the impact of the confounding factors, i.e., LOC, WMC, and code churn, we could notice that only code churn has a statistically significant relationship with all dependent variables. On the contrary, WMC turned out to have a poor impact on the security metrics, indicating that the complexity of methods is not related to the number of critical components in the source code. All in all, the ORs for all confounding factors are still very low, translating into a very weak effect on security.

Last but not least, we also found that in some cases, the projects themselves turned out to be significant for the explanation of the dependent variable. From a practical point of view, this means that the peculiarities of the projects have some influence on the changes to the global security profile. Our study cannot uncover the reasons behind this finding, as it would deserve further investigation. Yet, it is reasonable to believe that project-specific properties exist, e.g., contribution guidelines (Elazhary et al. 2019), code of conducts (Tourani et al. 2017), and more, that make developers more or less prone to introduce vulnerabilities.

In RQ₂, we investigated the relation between refactoring and security-related technical debt, measured through the number of security violations detected by SonarQube and the security remediation effort. Similarly to RQ₁, Table 7 reports the sign of the logit coefficients (within a circle) and the value of the ORs obtained for the Multinomial Log-Linear models for each refactoring type. Here too, the coefficients of the variables that turned out to be statistically significant are reported with a green or red , otherwise are left white. Whenever the impact of a refactoring type rejects a null hypothesis formulated in Section 2.5 the related cells are depicted in gray. The table does not report the confounding factors (LOC, WCM, and code churn), but these are reported in the raw results in our online appendix (Iannone et al. 2022).

Looking at the results, we could immediately notice the predominant presence of negative coefficients associated with both the decrease or increase of rule violations, implying that the majority of refactoring operations tend to keep the number of violations stable. This is particularly evident for Extract Interface, Move Package, Push Down Method, and Push Down Attribute. In other words, these refactoring types generally do not introduce or resolve any security-related violation. This is in line with their definition. Extract Interface and Move Package do not overhaul the code structure of the involved classes, so it is reasonable that they do not affect any security rule. The refactoring operations that push attributes or methods down to class hierarchies—i.e., Push Down Method, Push Down Attribute—do not seem to affect security rules in any way. However, both Move Package and Move Attribute are connected to a variation of the security remediation effort. This could be explained by the fact that these kinds of changes are made in conjunction with other changes that introduce security-related technical debt.

Despite these results, there are some notable exceptions worth analyzing. Extract Superclass refactoring significantly increases the chance of increasing the security remediation effort. Such a refactoring type also tends to violate rule S2386, i.e., ‘Mutable fields should not be public static’. Violating such a rule has a security implication, as the presence of public static fields expose mutable objects or arrays to changes by malicious users. Such bad practices are also categorized by CWE-582: ‘Array Declared Public, Final, and Static’, CWE-607: ‘Public Static Final Field References Mutable Object’, and CWE-766: ‘Critical Data Element Declared Public’, all representing weaknesses in the source code that attackers can leverage to carry out attacks. In other words, mutable objects should not be leaked to client programs as they can violate class invariants and disrupt the normal execution flow of the target application—if they have access to its runtime. Moreover, violations of this rule also imply a degradation of encapsulation, further motivating the importance of resolving it early. Curiously, an increase in security remediation effort also happens with Extract Interface refactoring. Despite both refactoring operations aiming to simplify the hierarchical organization of a project, they should be made with caution as they have a high risk of increasing the remediation effort. Let us consider an example in project batik at the revision e28370d2, also shown in Fig. 3. The commit applied a series of refactorings to reorganize some hierarchical structures in the util package—as also stated in one paragraph of the full commit description: ‘[...] Cleaned up, made easier to extend and pulled several inner class out of ParsedURL [...]’. In particular, the class URLData was first promoted from a static nested class to a first-level public class, renamed to ParsedURLData and then reorganized to have a new subclass called DataParsedURLData—hence, an Extract Superclass was applied. In the end, the ParsedURL class was streamlined to favor the new extension. In applying these changes, the former nested class had five public attributes left unchanged when the class was made public. This caused SonarQube to recognize a violation to rule S2386 for the five attributes, as now they can be freely modified by an external client program. In this case, we observe that the refactoring alone is not the direct cause of the violation, but the way it was applied led to the creation of extra public static fields. To conclude, Extract Superclass and Extract Interface refactorings should be applied with particular care as they have also been seen to disrupt all the security metrics (RQ₁).

Another exception occurs with Extract & Move Method refactoring, that (1) negatively impacts rule S2647, i.e., ‘Basic authentication should not be used’ and (2) increases the chances of rule S5547, i.e., ‘Cipher algorithms should be robust’ being removed—as it can be observed from its high OR. The individual refactoring operations, i.e., Extract Method and Move Method, do not appear to be connected with these rules in any form, while their combination has observable effects. Analyzing this case further, we could not identify specific reasons why the combination of multiple refactorings has a higher impact than individual refactoring types, yet we can suppose that our results represent a reflection of the number of changes applied, i.e., more changes affect security more than individual ones. Nonetheless, the effect of refactoring sequences is something that might be worth further analyzing in future work.

As in RQ₁, it is worth noting that the results were achieved while controlling for several confounding factors. Similarly to the previous discussion, the confounding factors are generally not statistically significant in any model, i.e., they are not correlated with the increase or decrease of security-related technical debt. Likewise, the projects turned out to be significant, somehow confirming that there exist some project-specific attributes that might influence the security technical debt.

Our third research question investigated the relationship between refactoring and the introduction of known vulnerabilities reported in the National Vulnerability Database (NVD).

It is worth recalling that for this research question, we focused on nine of the projects considered in the study (see Table 1). This subset of projects is affected by 26 known vulnerabilities, i.e., 26 different CVE records, whereas the number of distinct VCCs was 103—there were some cases of commits contributing to more than one vulnerability. Table 8 reports the descriptive statistics of the distribution of refactoring operations, grouped by type, in such VCCs. In the first place, our results showed that the number of VCCs with at least one refactoring was 34, i.e., 33.01% of the VCCs contained at least one instance of a refactoring operation. While this seems to indicate that refactoring might have a connection with the introduction of vulnerabilities, a closer look indicates a lack of causal relationship between the refactoring activities performed by developers and the introduction of vulnerabilities, i.e., the fact that refactoring is performed does not imply that it is the root cause of the vulnerability introduction.

A more in-depth analysis reveals that the refactoring types that occurred the most in the VCCs were Pull Up Method and Pull Up Attribute with 251 and 126 instances, respectively. Both refactoring types deal with generalization, hinting that complex restructuring activity (i.e., modifying hierarchies) are often present when vulnerabilities are introduced—this is partially in line with the results observed in the context of RQ₂.

Nonetheless, we also observed their very high standard deviation values that, combined with much lower mean values, imply that the distribution of these refactoring types across the commits is “irregular”—i.e., there are commits with a considerable number of Pull Up Method and Pull Up Attribute and commits without any of them. For instance, the Jenkins’s commit 70c10658 has over 200 instances of Pull Up Method and over 80 of Pull Up Attribute. Such a commit touches over 300 different files and represents a crucial commit for the project as it marks the moment when Jenkins was forked from Hudson (its original project). This suggests that some projects, like Jenkins, are characterized by large and poorly cohesive commits, which have higher chances to touch critical parts of the code, possibly introducing defects and security flaws. Hence, a “chaotic” development process might be the actual reason behind the introduction of vulnerabilities—as part of our future research agenda on the matter, we plan to investigate this aspect further.

From a different point of view, the nine NVD projects had a total of 7,708 refactoring commits (i.e., commits with at least one refactoring instance), 34 of which contributed to the introduction of a vulnerability, accounting for 0.44% of the total. This further supports the fact that refactoring alone is not the main responsible for the introduction of vulnerabilities, but rather a co-occurring phenomenon that, in some cases, might worsen the situation, especially when touching several components.

To assess whether the number of a specific refactoring type was statistically significant, we run a one-tailed Mann-Whitney U test (Mann and Whitney 1947) for each refactoring type on both the sets (i.e., the refactoring commits contributing to vulnerabilities versus those that did not contribute), for a total of 14 test runs. We discovered that the distribution of Extract Superclass and Extract & Move Method instances for VCCs is significantly higher than the distribution for non-VCCs. (p < 0.05). At the same time, Cohen’s d (Cohen 2013) is lower than 0.2, indicating a very small effect size. In other words, Extract Superclass and Extract & Move Method occurs more often in VCCs, but still in a limited way. Extract Method and Move Method refactorings do not appear to show any connection with VCCs, hence suggesting that basic refactorings touching few code components are less likely to contribute to the emergence of a vulnerability. On the contrary, only for Pull Up Method and Pull Up Attribute the effect size appeared large (d > 1.2), but without any statistically significant difference highlighted by the Mann-Whitney U test. The full results of such tests are reported in our online appendix (Iannone et al. 2022).

Going more in-depth, let us consider the example reported in Fig. 4. It shows the diff of the commit e45d7bda of the project Conversations, an XMPP client for Android that allows the creation of private chats with other users. The commit message states that a “UI code refactoring” was applied. The modification impacted three different files. Four years later, the modification resulted in being one of the causes that led to vulnerability CVE-2018-18467, which allowed an attacker to append a custom text to an existing conversation (with a draft message) by sending an intent from another application. While the commit message suggests code refactoring as the main activity performed, the vulnerability was not due to the refactoring itself, but rather to the addition of the appendText() method in the class ConversationFragment. This allowed appending any text to an existing conversation without adequately checking if an external application was trying to append a text content to an existing draft message through an Intent, leading to the vulnerability described in CVE-2018-18467. In other words, the vulnerability was involuntarily introduced while the committing author was doing some refactoring and code clean-up. In the example, RefactoringMiner only managed to mine two instances of Inline Method, which only represent a small part of the total modifications made. Thus, we can conclude that refactoring is often not the direct cause of vulnerabilities but rather a co-occurring phenomenon.

The results of our three research questions allowed us to quantify the role of refactoring on three critical aspects of software security, such as its impact on security metrics, technical debt, and introduction of known vulnerabilities. Moreover, the statistical analyses conducted enable a more general and conclusive discussion of the initial hypotheses formulated in Table 2.

As we learned from RQ₁, it is indeed possible that some refactoring operations may influence security metrics depending on how they are applied, while they rarely have an impact on technical debt (RQ₂) and introduction of known vulnerabilities (RQ₃); similarly, it may happen that a change accompanied by refactoring can contribute to a vulnerability without affecting any security metrics or increasing the technical debt value. This is the case of the example shown in Fig. 4 in which neither Surface nor SonarQube detected any difference between the commit containing the refactoring (e45d7bda) and its predecessor (i.e., the Δ was mapped to the “Stable” category). Hence, these results partially contradict the preliminary findings reported by Abid et al. (2020): when studying the matter on a larger scale, it comes out that most refactoring operations do not directly impact the security of software systems but are rather co-occurring phenomena.

However, some exceptions have been observed, especially when considering security technical debt. According to our results, the Extract Interface refactoring provides a significant increase in security-related technical debt but might have positive effects on some security metrics, e.g., reducing the number of classified methods (CM metric). This result supports and further stimulates the research efforts on the construction of automated refactoring recommenders that might balance quality improvements and security threats, as initiated by Abid et al. (2020).

To broaden the scope of the discussion, our overall findings do not match with the results previously obtained when studying the relation between refactoring and defects (Bavota et al. 2012; Di Penta et al. 2020). In particular, this is the case of the refactoring types dealing with the generalization: while Bavota et al. (2012) and Di Penta et al. (2020) found these operations to be sometimes defect-inducing, we discovered that they can instead provide some benefits to security aspects connected to attribute encapsulation. We see two main points here. On the one hand, these differences corroborate the conclusions drawn by previous researchers on the need of considering and treating vulnerabilities differently from defects (Canfora et al. 2020; Camilo et al. 2015; Joshi et al. 2015; Mercaldo et al. 2018; Morrison et al. 2018; Russo et al. 2019). On the other hand, our results indicate that the same refactoring can have multiple, contrasting effects on code quality and dependability.

The results of our study provide us with several actionable items and implications for both researchers and practitioners that we discuss in the following.

Many studies have investigated the impact of refactoring on software quality either directly or from the perspective of defect proneness, change proneness, or code smells.

Bavota et al. (2015) mined the history of 63 releases of Java Open Source Projects (OSPs) to investigate refactoring operations on code components, which indicate a need for refactoring through indicators such as metrics and smells. They concluded that most refactoring operations take place on code with no quality metrics indication for the need to refactor, and although 40% of refactoring operations were performed on smelly code, only 7% removed the smells. Their findings were corroborated by Yoshida et al. (2016) who revisited the relationship between code smells and refactoring by using the same refactoring dataset by Bavota et al. (2015). Cedrim et al. (2017) had similar findings when they analyzed more than 16K+ refactoring instances from 23 OSPs to investigate whether refactoring reduces the code smell density. They reported that even though almost 80% of refactorings touched smelly code, 57% of refactorings did not impact them, and roughly 10% of them removed the smells while 33% introduced new smells. Tufano et al. (2017) conducted an empirical study on 200 projects from the Android, Apache, and Eclipse ecosystems to investigate, among other aspects, whether developers’ actions, e.g., refactoring, resolve smells. They reported that only a low number (9%) of code smells are removed following refactoring operations.

Palomba et al. (2017) investigated the relationship between refactoring operations and code changes (namely fault repairing modification, general maintenance modification, and feature introduction modification) by analyzing the dataset by Bavota et al. (2015). They concluded that code duplication and Self-Admitted Technical Debt (SATD) are the main reasons behind refactoring instances, and refactoring also helps increase code readability. The impact of refactoring on code readability was the subject of the study by Sellitto et al. (2022), who partially confirmed previous findings, showing that refactoring can also negatively impact code readability metrics.

Kim et al. (2012) investigated the refactoring benefits and challenges at Microsoft by conducting a survey, semi-structured interviews, and historical data analysis. They found that refactoring is beneficial, leading to reduced inter-module dependencies and post-release defects. An empirical study was conducted by Bavota et al. (2012) to investigate the impact of refactoring on defects. They found that generally, refactoring instances do not induce defects. However, in specific cases, some specific refactoring types (e.g., Pull Up Method and Extract Subclass) tend to introduce defects in code.

Mumtaz et al. (2018) investigated whether removing code smells through refactoring resulted in improved security for a system. They conducted a study to identify a subset of code smells and calculated security metrics on five systems. Then they applied refactoring to remove the smells and then re-calculated the same metrics as before. They concluded that generally, refactoring improved the quality of the studied systems from a security standpoint. Ghaith and Cinnéide (2012) were interested in finding out whether automated search-based refactoring improved software security. They achieved an improvement of 15% in the metrics of industrial software after applying search-based refactoring. However, their study is based on a small project, and the results are not generalizable.

An empirical study was conducted by Abid et al. (2020) to determine the relationship between quality and security and the impact of refactoring types on security. The results of the study were used to implement a tool, which was then evaluated on OSPs. They concluded that their tool improved the security of the systems with little impact on the quality. They further validated their results by conducting a survey with practitioners. Similarly, Alshammari et al. (2010a) assessed the impact of refactoring at the design level on security using a case study. Their findings indicate that about 20 refactoring rules improve security, 12 rules made security worse, and four rules had no impact on security. In a follow-up study (Alshammari et al. 2012), the authors evaluated the impact of refactoring on information security using a case study. 8 out of 16 refactoring rules used improved the software’s security while the remaining made it worse. Again, both studies being focused on one case study cannot be generalized. Maruyama and Omori (2011) proposed a tool, implemented as an Eclipse plug-in, to help developers assess the impact of their refactoring operations on software vulnerabilities during software implementation. Currently, the tool supports only two refactoring types, namely, Pull Up Method and Push Down Method, and measure security using access levels (private, public, protected, and default) of fields. A downgrade in the access level signifies that the software becomes more vulnerable. However, they evaluated the tool using an artificial experiment (on one version of Eclipse) and not on real software.

Refactoring has been recognized in many studies as one of the most common ways to manage technical debt (Pérez et al. 2020; Codabux and Williams 2013; Codabux et al. 2014). In this subsection, we review some studies to understand the impact of refactoring on security debt (TD).

Zabardast et al. (2020) investigated the impact of various software development activities, including refactoring on TD by analyzing 2K+ commits in a large industrial project. Their empirical study shows that refactoring removes 22% of TD but introduces an additional 22% TD. However, in most cases, refactoring did not impact TD. Search-based automated refactoring using four different approaches was used by Mohan et al. (2016) to determine the impact of refactoring on TD, among other aspects of development, on six OSPs. They concluded that automated refactoring help decrease TD in software. However, these studies do not focus on security-related TD. Similarly, there are some studies on security smells, which are symptoms in the code that signals the prospect of a security vulnerability (Ghafari et al. 2017). Such studies investigate whether the security smells have an impact on vulnerabilities and are conducted in specific domains (Ghafari et al. 2017; Rahman et al. 2019).

To summarize, the studies that investigate the impact of refactoring on software quality have mixed results. Some reported a positive impact whereas others concluded that refactoring increased TD or had no impact on it. Most existing studies focus on the impact of refactoring on software quality but very few investigate TD specifically. Similarly, the studies which investigate the impact of refactoring on software security do not include security-related technical debt. The studies are also limited, often focusing on one software system, thereby making their results not generalizable. Despite refactoring being commonly used to reduce technical debt, most existing research focuses on code smell as an indication of debt, thereby explaining the lack of refactoring studies that focus on technical debt directly. Similarly, security smells have been investigated in the context of vulnerabilities but not refactoring. Abid et al. (2020) conducted a preliminary study to investigate the impact of refactoring types on security but, to the best of our knowledge, investigating the actual impact of refactoring on technical debt from a security standpoint has not been studied and therefore represents a premier of our research.