Simulating all archetypes of SQL injection vulnerability exploitation using reinforcement learning agents

This work further develops the contribution of Erdodi et al. [6], and Del Verme et al. [7], which only handled union-based vulnerabilities without considering other vulnerability types. Our main contributions are as follows. (1) We extend this approach to all the archetypes of SQL injection vulnerabilities. (2) In order to make the simulation more realistic, our work is the first not to assume that the simulated website is vulnerable. This forces the agent to investigate the website more thoroughly and come to a decision, as penetration testers would do, on whether the website is vulnerable to SQL injection exploitation or not. (3) We simplified the preprocessing. In previous works, human input was necessary to convert server responses to meaningful inputs for the agent. We show that our model can deal with the raw message by relying on features of the HTML response, such as its length or keywords. This also gives the agent higher control, potentially allowing future agents to make more interesting inferences and decisions. (4) Last, this work also shows that the agent exhibits a degree of transfer learning, meaning that it does not learn to solve each SQL vulnerability in isolation, but it can use queries to gain information about different types of SQL injection vulnerabilities. In conclusion, we showcase more performant RL agents that can tackle a wider variety of SQL injection-related problems.

The rest of the paper is organized as follows. Section 2 contains background information, as well as a review of the literature. Section 3 covers the details of our RL problem. After that, we list and discuss our results in Sect. 4. Finally, in Sect. 5, we conclude our study and list possible future work.

In the world of ethical hacking and penetration testing, CTF challenges are often used to measure the skill and ability of an expert or a team of experts. A committee sets up a mock environment containing potential vulnerabilities, and one or more teams of experts are tasked with identifying the vulnerabilities and hacking them; a flag hidden behind each vulnerability is used as proof of a successful exploitation. Upon submission of a valid flag, a team is awarded points for completing the challenge. This setup is straightforward, flexible, and conforms well to a typical RL problem in which an agent interacts with an environment and receives rewards for completing its tasks.

Most websites are supported by a database, accessed based on some user input. For example, when using a login form, often there is a SQL query accessing the database to check the validity of the entered username and password. This query is typically hidden from the user. If this hidden query¹ does not have proper input validation, the website has an SQLi vulnerability. In this case, a malicious user can craft an input that would allow her to alter the interpretation of the SQL query in order to execute customized requests and potentially obtain protected information.

Discovering and exploiting a SQLi vulnerability is hard since the attacker has limited information on what is happening server-side. A first step usually includes an exploratory phase during which the attacker sends requests and analyzes responses in order to learn more about the target system, for instance:After this preparatory phase of information gathering, the attacker may try to perform a SQLi exploitation. Based on what she has learned, the attacker may identify a specific SQLi scenario; five archetypical vulnerability exploitation scenarios are:Notice that, in addition to these five scenarios, the attacker may also decide that the target server exhibits no vulnerability. The above exploitations differ both in the information that an attacker can collect during the exploratory phase (informative responses, binary outputs, error messages, time information) and the actual form of exploitation (execution of arbitrary code, information disclosure).

RL is a ML method for solving optimization problems in dynamic environments. A RL problem is expressed as an agent acting in an environment [8]. The goal for this agent is to learn an optimal policy that maximizes its goals or its satisfaction. The agent can take actions in the environment, observe changes, and collect rewards. By inferring the relationship between its actions and the reward amount, the agent reinforces those behaviors that led to high rewards.

Formally, a RL problem is defined by a tuple \(({\mathcal {S}},{\mathcal {A}},{T},{R})\) constituting a Markov Decision Process (MDP), where:In this setup, the objective of the agent is to find the action policy \(\pi \) that maximizes the sum of its rewards in the long term, that is:where t is a time-index, T is the time horizon of the problem, \(\gamma \) is a discount factor that favours immediate rewards over rewards far in the future, \(r_t\) is the reward obtained at step t, and \(E[\cdot ]\) is the expected value [9].

Practically, an agent interacts with an environment for a number of episodes e, each one consisting of T steps; during these interactions, it learns and finetunes its policy \(\pi \) using a RL algorithm.

A well-known RL algorithm to train an agent is Q-learning; this algorithm joins conceptual simplicity with good performance. In its tabular version, the Q-learning algorithm requires the instantiation of a matrix tracking the value of each action in each state. The agent updates this matrix by changing the value of the current state (\(S_t\)) as it is about to enter a new state (\(S_{t+1}\)), based on the action chosen (\(A_t\)) and the reward received (\(R_t\)). Formally, this can be expressed as:where Q is the action-value matrix and \(\alpha \) is the learning rate that regulates the magnitude of the updates [9]. By looking up its Q-table, the agent can easily decide the best action in a given state. During learning, however, the agent has to balance between exploitation (selecting an action currently considered the best) and exploration (trying out a random action to assess its consequences). This trade-off can be heuristically solved by introducing an exploration rate parameter, \(0 \le \epsilon \le 1\), which, at each step, gives the probability for the agent of selecting a random action. This allows an agent to explore more and lessens the chance of being stuck in a local optimum. In large or infinite state spaces, a Q-learning agent cannot explore the whole space; in this case, it aims to find an acceptable sub-optimal policy that allows it to achieve satisfactory results.

SQLi is a well-documented vulnerability [1, 2], and several automatic tools have been developed to help with prevention. Web vulnerability scanners focus on detecting SQLi vulnerabilities relying on predefined requests; tools, such as Acunetix [10], typically send out a few requests for each web input parameter and observe the response to hypothesize whether a vulnerability is present. Vulnerability scanners only perform detection; they are not designed to exploit the vulnerability. This makes vulnerability scanners prone to false positives, as their hypotheses cannot be verified through an actual attempt at exploitation. Exploitation tools, instead, aim at performing exploitations; SQLi specific tools, such as SQLmap, sqlninja, or pangolin, can exploit the vulnerability using a very well-defined attack logic. These tools are also not perfect because they are bound to a pre-encoded logic, and they can produce many false negatives requiring the review of a human expert. State-of-the-art tools for SQLi thus still require the supervision of an experienced human pentester. Taking the human expert out of the loop would require a solution that still uses predefined requests (as the current tools do) but can also improve its performance by learning from previous cases and occasionally exploring new possibilities, as human testers do.

Our approach to solving the SQLi problem by using ML can be placed in the larger field of ML for cyber security. ML for cyber security has been recently used primarily for defensive operations and less for offensive ones [11, 12]; similar to our work, some researchers have considered RL for defensive operations [13, 14]. Our research, however, focuses more on a subgroup of RL for offensive operations, similar to works [6, 7, 15‐17]. In [15] Bland et al. deploy RL agents in an environment modeled with the extended Petri net formalism put forward by [18]; RL agents take on the role of either attacker or defender. Zennaro and Erdodi [16] exemplify the offensive cyber capabilities of RL agents for pentesting. Ghanem and Chen [17] reintroduce and extend upon their previously introduced IAPTS system [19] which allows RL agents to learn from real-world data with the aid of expert penetration testers in order to speed up and improve manual penetration testing; they show promising results on their experiments. Our work on SQLi addresses a specific subset of penetration testing challenges; however, this is not the first work to use RL for SQLi. Erdodi et al. successfully applied RL on a simplified union-based vulnerability [6]; they did so with a highly structured approach. A year later, Del Verme et al. proposed a less structured approach with the potential of finding unanticipated SQL vulnerabilities [7]. However, the unstructured approach appears computationally expensive; this is part of the motivation behind the adoption of a more structured approach in this work. This work exploits different SQL injection vulnerabilities while [15] focuses on Cross Site Scripting and Phishing. [16‐19] present a general approach for penetration testing with RL without focusing on any unique exploitation and HTTP level requests. Other works with RL and SQLi like [7] have a different approach by creating the SQL commands without any prior SQL knowledge. [6] exploits only the union-based vulnerabilities. This research takes the RL-based SQL injection exploitation a significant step forward by targeting all SQLi vulnerability archetypes and building the attack from simple web request actions.

Like any penetration testing or ethical hacking tool, an autonomous SQLi agent that identifies and exploits vulnerabilities bears the risk of misuse. The authors want to remark that this work is aimed at testing and improving the security of potentially vulnerable websites and condemn the agent’s use in illegal or unethical activities.

We implemented a dynamic mock-up environment in which, at each instantiation, a server with zero or one SQLi vulnerability is initialized. The environment receives the RL agent’s requests, processes them, and returns a response. The learning agent is expected to interact with the server, detect what sort of vulnerability is present, and then provide the information necessary to start an automated exploit. Each type of SQLi vulnerability has a different type of condition of success.

Our action space consists of 65 possible discrete actions divided into 38 predefined queries, one action to declare the absence of any SQLi vulnerability, and 26 exploit actions consisting of multiple queries. Each action serves a specific purpose, as explained below (the exact definition of all the actions is available in “Appendix A”):The exploit actions consist of many queries. The number of queries depends on the attacker’s goal and which type of vulnerability is being exploited. By choosing this action, the agent has determined the vulnerability and has, at this point, all the information necessary to carry out the attack. However, for computational and practical reasons, we do not require the completion of the attack. An attacker may cause an information disclosure, modify one or multiple data in the database, or even, in more advanced cases, try to execute OS commands; our agent stops short of performing the attack.

Importantly, all the actions above have no pre-defined meaning for the agent. Even though we partition them into well-defined subsets, all actions are identical syntactical constructs for the agent. Through interaction and trial-and-error, the agent infers each action’s relevance and builds a strategy to tackle the different possible challenges it has to confront.

The state is used to represent what the agent knows, while state transitions are used to capture the evolution of the knowledge of the agent. Therefore, we let the state be a set of sets, where each set contains all the actions resulting in the same message m. Most sets, thus, contain all actions that resulted in an HTML response with the same length. For example, if the state is ((2,), (3, 5, 6))), it means that action number 3, 5, and 6 resulted in the same length, while query number 2 resulted in a different length. Since we have some unique responses, such as \(-2\) and \(-3\), we have special sets for containing the responses associated with them. This means that the example above would read as ((\(-3\),), (\(-2\),), (2,), (3, 5, 6),). In this case, it would mean that our \(-3\) set is empty and our \(-2\) set is empty, but 3, 5, and 6 resulted in the same length, but 2 resulted in a different length.

In these proof-of-concept simulations, we use tabular Q-learning. Tabular Q-learning was chosen over alternative reinforcement learning algorithms, such as PPO [20] and DQN [21], because of its simplicity and interpretability. Interpretability is particularly important in our simulations as it allows us to analyze, monitor and explain the agent, both once it is trained and during training. This allowed for a more in-depth analysis of how the agent simultaneously tackles the different SQL injection vulnerability exploits. This technique could be extended to work with other reinforcement learning algorithms, but this is outside the scope of this work.

Simulation1 considers the first three vulnerability types: stack-based, union-based and Boolean-based blind, along with the possibility of interacting with a server with no vulnerability. In this simulation, error and time information are irrelevant; only the HTML page length and SQL version (\(m=-2\)) are relevant. All environment pre-processing boils down to identifying a pre-defined string (e.g., 8.0.21-0ubuntu0.20.04.4) or computing the length of the server HTML response. Finding the version number gives our agent definitive proof that it can access secret information. Therefore, it confirms that it has found the vulnerability before it executes its final action, which can be very costly as it consists of multiple queries. For an expert, the SQL version also tells her which exploits may be infeasible; the SQL version may protect against some vulnerabilities.

This simulation represents a small extension of the existing state of the art. Instead of considering a single type of SQLi vulnerability, the agent is presented with three different vulnerabilities. These vulnerabilities share some similarities, and we expect the RL agent to be able to exploit these commonalities in learning and to effectively exploit stack-, union-, and Boolean-based blind vulnerabilities. As a baseline, we would expect the agent to use around four queries: two to find the correct escape out of three possible options, one to distinguish the right vulnerability, and another to declare the start of the exploitation. According to a sequential approach commonly used by pentesters, we also expect the agent to first master identifying no vulnerability as this is simpler than exploiting a vulnerability. Next, we expect the agent to exploit stack-based vulnerabilities as it only needs to find the escape. Then exploit Boolean-based blind as it needs the escape and the Boolean-value of the hidden query, and finally, union-based vulnerabilities as it needs to find the escape and the number of rows.

Simulation2 adds to Simulation1 error-based vulnerabilities, thus bringing the number of possible scenarios the agent faces to five (four vulnerabilities plus absence of vulnerability). Our environment now performs more preprocessing to capture errors displayed to the user and produce the message \(m=-3\); practically, this requires the pre-processor to capture error pages. For simplicity, we assume the target website not to contain the word error, and so we reduce the preprocessing only to detecting the presence of the word error.

Adding a new vulnerability increases the challenge for the agent. However, in our encoding, error-based vulnerabilities are relatively easy to identify and exploit, so we do not expect a significant change in the learning dynamics or a large increase in the number of states explored by the agent. Actually, after successfully learning, we may expect the average number of queries to decrease since an error-based vulnerability would require only two steps to exploit. In a sequential approach, we expect the agent to get a handle on error-based vulnerabilities and no vulnerability cases first, then follow the same pattern as in Simulation1: solving stack-based, Boolean-based blind, and finally union-based vulnerabilities.

Finally, in Simulation3, we include all vulnerabilities. Adding time-based blind vulnerabilities requires the environment to return the time information along with the message m. For simplicity, response time is divided into a long and short time, chosen with an arbitrary threshold. In general, when the query is true, the server will use more time to access the relevant archives, and the environment will generate a long-time message. On the other hand, if the query is false, the response will come quickly and generate a short-time message. However, the time message is not deterministic because of network traffic. With a certain probability, p high-network traffic will change a short-time message into a long-time message. We experiment with a deterministic no-traffic network (\(p=0\)) and a stochastic low-traffic network (\(p=.05\)). The presence of traffic will introduce uncertainty, making it reasonable for the agent to retry its queries from time to time. However, given low traffic and the availability of multiple queries that can be used to assess the target, we find that the agent may not need to store duplicate query results. Despite the probability of traffic is low, the likelihood of an episode being affected by an event of high-traffic increases with the number of queries. The relationship between the episode traffic probability, \(P(e_t)\), and the query traffic probability, \(P(q_t)\) is given by:where n is the number of queries in the episode; notice that in our computation we consider \((n-1)\) assuming the last query to be the exploitation, after which we are not concerned with network traffic anymore. This means that episodes that end in 1, 2, 10, and 20 queries have an episode traffic probability of 0%, 5%, 37%, and 62.3%, respectively.

The addition of time-based blind vulnerabilities, especially in the presence of traffic, significantly increases the complexity. They represent a vulnerability having fewer commonalities with the previous ones, thus limiting the amount of information the agent can transfer during learning. We expect the need for more extensive training and a modest increase in the number of states explored during training. In the absence of traffic, the agent could solve the time-based blind vulnerability similarly to Boolean-based blind problems, as both problems provide binary information, either a simple true or false or as a long-time and short-time. With traffic, however, time-based blind will be require more time to be solved.

We train the agents in our dynamic SQL environment for a million episodes, except for Simulation3, where we increased it to ten million. The first 90% of the episodes adopt an exploration rate of 0.1 except for Simulation2, which has an initial exploration rate of 0.02, while the last 10% use an exploration rate of 0. An episode terminates if the agent executes the correct exploit action or claims that there is no vulnerability. Alternatively, we set a maximum number of queries per episode to 100. Notice that this number is larger than the set of available actions. This limit is meaningful only at the beginning when the agent will randomly explore all the possible actions and observe their results. By the end, however, this limit will be irrelevant as the agent will have learned a strategy to achieve its goal with a few steps. We let the reward of a single query be \(-1\); the flag returns a value worth 1000, and an incorrect solution attempt, including deciding to give up when there is a solution, produces a reward of \(-1000\).

We further evaluate the behavior of our agents by testing them on 1000 vulnerabilities logging how many actions they take to exploit a vulnerability and whether or not they were successful in the exploiting the vulnerability.