\(\hbox {CDBTune}^{+}\): An efficient deep reinforcement learning-based automatic cloud database tuning system

\({\texttt {CDBTune}}^{+}\) first trains a model based on some initial training data. Then, given an online tuning request, CDBTune utilizes the model to recommend knob settings. CDBTune also updates the model by leveraging the tuning request as training data.

Figure 2 illustrates the architecture of \({\texttt {CDBTune}}^{+}\). The dotted box on the left represents the client, where users send their tuning requests to the server through the local interface. The other dotted box represents our tuning system in the cloud, in which the controller of the distributed cloud platform coordinates the client, the CDB and \({\texttt {CDBTune}}^{+}\). When the user initiates a tuning request or the DBA initiates a training request via the controller, the workload generator conducts stress testing on the CDB instances which remain to be tuned by simulating workloads or replaying the user’s workloads. At the same time, the metrics collector collects and processes related metrics. The processed data will be stored in the memory pool and fed into the deep RL network respectively. Finally, the recommender outputs the knob configurations which will be deployed in the CDB.

Both the search-based approach and the multistep learning-based approach suffer from some limitations, so we desire to design an efficient end-to-end tuning system. At the same time, we want our model to learn well with limited samples during initial training and to simulate the DBA’s train of thought as much as possible. Therefore, we tried the RL method. At the beginning, we tried classic Q-learning and DQN models in RL, but both of these methods failed to solve the problems incurred by our high-dimensional space problem (database state, knobs combination of knobs) and continuous actions (continuous knobs). Eventually, we adopt the policy-based Deep Deterministic Policy Gradient approach which overcomes these shortcomings effectively. In addition, the design of the reward function (RF) is vital (as the “soul” of RL), which directly affects the efficiency and quality of the model. Thus, by simulating the DBA’s tuning experience, we design a reward function which is more in line with tuning scenarios, and makes our algorithm effective and efficient. RL uses the exploration & exploitation strategy [27] to potentially explore more optimal configurations that the a DBA would never have tried. For example, when training our model, we randomly output a set of recommended configurations with a small probability (e.g., 0.1) and follow an output according to the network learning strategy with a large probability (0.9). This allows us to explore the unknown tuning space with a small probability so as to reduce the possibility of falling into a local optimum. Although the exploration in RL will not result in arbitrarily bad performance from the users’ side (because we employ the training and tuning process first to backup databases), we are working hard to explore techniques that could constrain the recommended configuration to a “safe” range. This is an interesting and challenging question for future work.

The main challenge of using RL in CDBTune is to map database tuning scenarios to appropriate actions in RL. In Fig. 3, we show an interaction diagram of the six key elements in RL and the correspondence between the six elements and database configuration tuning.

Agent The agent can be seen as the tuning system \({\texttt {CDBTune}}^{+}\) which receives a reward (i.e., the performance change) and a state from the CDB, and updates the policy guiding the system to adjust the knobs for getting a higher reward (higher performance).

State The state denotes the current state of the agent, i.e., the 63 metrics. Specifically, when \({\texttt {CDBTune}}^{+}\) recommends a set of knob settings and the CDB applies them, the internal metrics (such as counters for pages read to or written from disk collected within a period of time) represent the current state of the CDB. In general, we describe the state at time t as \(s_t\).

Reward The reward is a scalar \(r_t\) which denotes the difference between the performance at times t and that at \(t-1\) or the initial settings, i.e., the performance change after/before the CDB applied the new knob configurations that \({\texttt {CDBTune}}^{+}\) recommended at time t.

Action An action originates from the space of knob configurations, which is often described as \(a_t\). An action here corresponds to a knob tuning operation. The CDB applies the corresponding action according to the latest policy under the corresponding state of the CDB. Note that an action might increase or decrease several tunable knobs’ values at a time.

Policy The policy \(\mu (s_t)\) defines the behavior of \({\texttt {CDBTune}}^{+}\) at a certain specific time and in a certain environment, It maps a state to an action. In other words, given a CDB state, if an action (i.e., a knob tuning) is called, the policy outputs the next state by applying the action to the original state. The policy here is a deep neural network, which contains the input (database state), output (knobs), and transitions among different states. The goal of RL is to learn the best policy. We will introduce the details of the corresponding deep neural network in Sect. 4.

RL-Based learning. The learning process of DBMS configuration tuning in RL is as as follows. The CDB is the target that we need to tune, which can be regarded as the environment in RL, while the deep RL model in \({\texttt {CDBTune}}^{+}\) is considered to be the agent in RL, which is mainly composed of a deep neural network (policy) whose input is the database state and whose output are the recommended configurations corresponding to the state. When applying the recommended configurations to the CDB, the current state of the database will change, which is reflected in the metrics. The internal metrics can be used to measure the runtime behavior of a database corresponding to the state in RL, while external metrics can evaluate the performance of a database for calculating the corresponding feedback reward value in RL. The agent will update its network (policy) according to these two pieces of feedback in order to recommend better performing knobs. This process iterates until the model converges. Ultimately, the most appropriate knob settings will be exposed. Note that we think \({\texttt {CDBTune}}^{+}\) is not database specific if the objective optimization system (e.g. a storage system, Spark, Hive and Tomcat) can be abstracted to the six elements mentioned above. This is an interesting question for future work to verify.

RL makes a policy decision through the interaction process between agent and environment. In contrast to supervised learning or unsupervised learning, RL depends on accumulated rewards, rather than labels, to perform training and learning. The goal of RL is to optimize its own policy based on the reward of the environment by interacting with the environment and achieving higher rewards by acting according to the updated policy. The agent is able to discover the best action through a trial-and-error strategy by either exploiting current knowledge or exploring unknown states. The learning of our model follows the two rules that the action depends on the policy, and that the policy is driven by the expected rewards of each state. RL can be divided into two categories: value-based method and policy-based methods. The output of the value-based method is the value or benefit (generally referred to as Q-value) of all actions and it chooses the action corresponding to the highest value. Differently, the output of the policy-based method is a concrete policy instead of a value and we can immediately output the action according to the policy. Since we need to use the actions, we adopt the policy-based method.

Q-Learning It is worth noting that Q-Learning [34] is one of the most popular value-based RL methods, at whose core is the calculation of Q-tables, which are defined as Q(s, a). The rows of the Q-tables contain the Q-value of the states while the columns of the Q-table represent actions, which measures how beneficial it will be if the current state is followed by this action. Q(s, a) is iteratively defined as follows:The basis for updating the Q-table is the Bellman Equation. In Eq. (1), \(\alpha \) is the learning rate, \(\gamma \) is a discount factor, which pays more attention to short-term reward if close to zero and concentrates more on long-term reward when approaching one, and r is the performance at time \(t+1\).

Q-Learning is effective in a relatively small state space. However, it is not well suited to solve problems with a large state space such as AlphaGo which contains as many as \(10^{172}\) states, because a Q-table can hardly store so many states. In addition, the states of a database in a cloud environment are also complex and diverse. For example, suppose that each inner metric value ranges from 0 to 100 and its value is discretized into 100 equal bins. Then 63 metrics will then have \(100^{63}\) states. As a result, applying Q-Learning to database configuration tuning is impractical.

DQN Fortunately, the Deep Q Network (DQN) [36] method is able to solve the problems mentioned above effectively. DQN uses neural networks rather than Q-tables to evaluate the Q-value, which fundamentally differs from Q-Learning (see Fig. 4). In DQN, the input are states while the output are the Q-values of all actions. Nevertheless, DQN still adopts Q-Learning to update the Q-value, so we can describe the relationship between them as follows:where \(\omega \) of \(Q(s, a, \omega )\) represents the weights of the neural network in DQN.

Unfortunately, DQN is a discrete-oriented control algorithm, which means that the actions it outputs are discrete. Taking the maze game for example [66], only four directions of output can be controlled. However, knob combinations in a database are high-dimensional and the values for many of them are continuous. For instance, if we use 266 (the maximum number of knobs that the DBA uses to tune a CDB) continuous knobs which range from 0 to 100. If each knob range would be discretized into 100 intervals, there would be 100 values for each knob. Thus there would be \(100^{266}\) actions (knob combinations) in total for DQN. Further, if we increase the number of knobs or decrease the learning interval, the scale of outputs would increase exponentially. Thus, neither Q-Learning nor DQN can solve the issue of database tuning. Thus we introduce the policy-based RL method DDPG to address this issue in Sect. 4.

We illustrate DDPG for CDBTune in Fig. 5. When utilizing DDPG in \({\texttt {CDBTune}}^{+}\), firstly, we regard the CDB instance to be tuned as the environment E, and our tuning agent can obtain normalized internal metrics \(s_{t}\) from E at time t. Then our tuning agent generates the knob settings \(a_{t}\) and will receive a reward \(r_{t}\) after applying \(a_{t}\) to the instance. Analogous to most policy gradient methods, DDPG has a parameterized policy function \(a_t = \mu (s_t|\theta ^{\mu })\) (\(\theta ^{\mu }\), mapping the state \(s_{t}\) to the value of action \(a_{t}\) which is usually called an actor). The critic function \(Q(s_t, a_t|\theta ^Q)\) of the network (where \(\theta ^Q\) denotes the learnable parameters) aims to represent the value (score) for a specific action \(a_{t}\) and state \(s_{t}\), which guides the learning of actor. Specifically, the critic function helps to evaluate the knob settings generated by the actor according to the current state of the instance. Inheriting the insights from the Bellman Equation and DQN, the expected Q(s, a) is defined as:where the policy \(\mu (s)\) is deterministic, \(s_{t+1}\) is the next state, \(r_t = r(s_t, a_t)\) is the reward function, and \(\gamma \) is a discount factor which denotes the importance of the future reward relative to the current reward. When parameterized by \(\theta ^Q\), the critic will be represented as \(Q^{\mu }(s, a|\theta ^Q)\) under the policy \(\mu \). After sampling transitions \((s_t, r_t, a_t, s_{t+1})\) from the replay memory, we apply the Q-learning algorithm and minimize the training objective:whereThe parameters of the critic are updated with gradient descent. As for the actor, we will apply the chain rule and update it with the policy gradient derived from \(Q(s_t,a_t|\theta ^Q)\):

The algorithm contains seven main steps, which are summarized in pseudo code in Algorithm 1. Note that we use the prioritized experience replay (PER) [43] method to accelerate our model convergence in online training (detailed see Sect. 5.1).

To make it easier to understand and implement our algorithm, we elaborate the network structure and specific parameter values of DDPG in Table 2. Note that we also discuss the impact of the recommended configurations by different networks on the system’s performance in Sect. 6.1.5.

Remark. Traditional machine learning methods rely on a massive amount of training samples to train the model. In contrast to that, we adopt the trial-and-error method to make our model generate diversified samples and learn via deep reinforcement learning, which only requires a limited amount of samples. We summarize the advantages of this approach. Firstly, for solving traditional game problems, we cannot predict how the environment will change after taking an action, because the environment of the game is random (for example, in Go, we do not know what the opponent will do next). However, in our DBMS tuning scenario, after a configuration is executed, the database (environment) will not randomly change after a configuration changes, due to the dependencies between knobs. Because of this, with relatively few samples, it is easier to learn DBMS tuning models than game problems with an adversary. Secondly, our \({\texttt {CDBTune}}^{+}\) model, only requires few input and output dimensions (63 and 266), which allows the network to efficiently converge without too many samples. Thirdly, RL also requires diverse samples, not only massive samples. For example, RL solves the game problems by processing each frame of the game screen to form the initial training samples. The time interval for each frame is very short, leading to a high redundancy of the training images. On the contrary, for DBMS tuning, we will change the parameters of database and collect the performance data. These data are diverse in our learning process, which allows us to constantly update and optimize the performance. Lastly, coupled with our efficient reward function, our method performs effectively with a small number of diverse samples (which will be evaluated in Sect. 6.3.1).

In summary, the DDPG algorithm makes it feasible for deep neural networks to process high-dimensional states and generate continuous actions. DQN is not able to directly map states to continuous actions for maximizing the action-value function. In DDPG, the actor can directly predict the values for all tunable knobs at the same time without considering the Q-value of a specific action and state.

The reward function is vital in RL, as it determines the feedback information between the agent and environment. We aim for a function that simulates a DBA’s empirical judgment of a real environment in the tuning process.

Next we describe how \({\texttt {CDBTune}}^{+}\) simulates a DBA’s tuning process to design reward functions. First, we introduce a DBA’s tuning process as follows: Based on the above idea, we aim to mimic the tuning method of DBAs, which not only considers the change of performance compared to the previous step but also to the initial state (the time when the decision to tune the database was made). Formally, let r, T and L denote reward, throughput and latency. Especially, \(T_0\) and \(L_0\) respectively denote the throughput and latency before tuning. We design the reward function as follows.

At time t, we calculate the rate of the performance change \(\varDelta \) from time \(t-1\) and the initial time to time t respectively. The detailed formula is shown as follows:According to Eqs. (6) and (7), we design the reward function below:\(\varDelta \) can refer to the performance change of latency L or throughput T. As the ultimate goal of tuning is to achieve better performance than the initial settings, we need to reduce the impact of the intermediate process of tuning when designing the reward function. We therefore set r to 0 if the result of Eq. (8) is positive and \({\varDelta }_{t \rightarrow t-1}\) is negative.

We calculate the reward for throughput \(r_{T}\) and latency \(r_{L}\) according to Eq. (8). We combine these two rewards with the corresponding weights \(C_{L}\) and \(C_{T}\), where \(C_L+C_T=1\). Note that these weights can be set based on user preferences. We have r to denote the sum of rewards of throughput and latency:If the goal of optimization is throughput and latency, our reward function does not need to change, because the reward function is independent of the hardware environment and workload changes, as it only depends on the optimization goal. Thus, the reward function would only need to be redesigned if the optimization goals were changed.

Note that other reward functions can be integrated into our system as well. We evaluate our designed reward function for the training and tuning process, by comparing it with three other typical reward functions in Sect. 6.3.1. Moreover, we explore how the two weights \(C_{L}\) and \(C_{T}\) will affect the performance of the DBMS in Sect. 6.3.2.

We briefly summarize the advantages of our method. (1) Limited Samples. In the absence of high-quality empirical data, accumulating experience via a trial-and-error method reduces the cost of data acquisition. We have our model generate diverse samples and learn towards an optimizing direction by using deep RL which only requires a limited amount of samples to achieve great effects (see Sect. 6). (2) High-dimensional Continuous Knobs Recommendation. The DDPG algorithm recommends better configurations in high-dimensional continuous space than simple regression methods, which are for example used by OtterTune (see Sect. 6.2.2). (3) End-to-End Approach. Our end-to-end approach reduces the potential for errors caused by multiple segmented tasks and improves the precision of recommended configuration. Moreover, the importance of different knobs is treated as an abstract feature which is implicitly learned by our deep neural network instead of us having to apply an extra method to rank the importance of different knobs (see Sect. 6.2.2). (4) Reducing the Possibility of a Local Optimum. \({\texttt {CDBTune}}^{+}\) may not find the global optimum, but RL adopts the well-known exploration & exploitation strategy, which is designed to efficiently explores configurations that a DBA may never try, thereby reducing the possibility of getting stuck in a local optimum (see Sect. 6.2.3). (5) High Adaptability. In contrast to supervised or unsupervised learning, RL has the ability to learn as much as possible in a reasonable direction from experience rather than from given examples, with a much lower dependency on labels or training data and shows a much higher adaptability to different workloads and hardware configurations in a cloud environment (see Sect. 6.4).

Successful attempts obtained during reinforcement learning are rare, resulting in an imbalance of positive and negative transitions. If a transition is randomly sampled in step 1 of Algorithm 1, failed attempts will be sampled with a high probability, which results in low efficiency of learning. How can we effectively prioritize the transitions we need to learn in DDPG? Prioritized experience replay (PER) is a method that is widely used in DQN. It is based on the idea to more frequently replay experiences associated with very successful attempts or extremely awful performance. Therefore, we adopt this method in DDPG. The so-called temporal difference error (TD-error) is used to update the estimate of the critic function Q(s, a) in DDPG. The computation of the TD-error \(TD_{m}\) of a transition m is given as:\(V_m\) is the Q-value/score of the current state and \(V^\prime _m\) is the estimated oneThe TD-error value is the loss of the critic network (which can be described as “how far the Q-value is from its next bootstrap estimation”) which reflects a correction for the estimation and may implicitly reflect to what extent an agent can learn from the experience. The larger the TD-error value is, the more space for improvement is still present in the prediction accuracy of the model, and the more important it is to learn from this transition. Replaying these transitions more frequently will help \({\texttt {CDBTune}}^{+}\) to gradually realize the true consequence of the wrong behavior in the corresponding states as well as avoid applying the wrong behavior in these conditions again, which can improve the overall database tuning performance. However, using the TD-error as the metric to sample the transitions will result in some transitions with very small TD-error values that are never sampled for learning. Therefore, we define the probability of the sampled transition m via the SoftMax:where \(p_{m} = (\mathrm{rank}(m))^{-1}\), and rank(m) denote the rank of the transition m in the experience replay memory with absolute TD-error being the criterion. Besides, the parameter \(\epsilon \) is the parameter which controls to what extent the prioritization is used. (Note that setting \(\epsilon \) to 1 corresponds to random sampling). This sampling strategy ensures that even transitions with a low TD-error have a chance of being sampled, which increases the diversity of sampled transitions and regularizes the model. Last but not least, since we tend to more frequently replay the transitions with a high TD-error, we violate the assumption that the states in reinforcement learning are accessed randomly. This may lead the convergence problems for \({\texttt {CDBTune}}^{+}\). In order to solve this problem, importance-sampling weights are used in the computation of weight changes:where N is the number of transitions and \(\beta \) is the parameter, which controls to what extent the correction is used. According to the above description. The Prioritized Experience Replay method in DDPG (PER-DDPG, Step 1 in Algorithm 1) used in our \({\texttt {CDBTune}}^{+}\) training process (and following our outlined approach) is shown in Algorithm 2. We also have evaluated the effect of this method on training \({\texttt {CDBTune}}^{+}\) in Sect. 6.1.1.

Changes in some of the tunable knobs in a CDB require a restart the database system. The restart time ranges from a few minutes to ten minutes because most database systems follow the classic Algorithms for Recovery and Isolation Exploiting Semantics (ARIES) [37], which typically create checkpoints periodically and store this information in a log file. When a database needs to restart, the system first needs to conduct the redo recovery from the last checkpoint to restore the data to the state before the restart (storage tier), then apply the undo recovery based on the undo log records to complete uncommitted transactions (compute tier). Note that a large amount of data I/O, file I/O, binlog I/O and redo log I/O operations are required to interact between the two tiers in this process. Therefore, the restart of a database system is a very time consuming operation.

To solve this problem, cloud databases draws on the idea of “The log is the database” from Aurora [56], PolarDB [8] and AnalyticDB [65] The differences between cloud databases and general database are shown in Fig. 6. CDBs differ in three characteristics from a general database: (1) they decouple the storage tier from the compute tier; (2) they offload all other types of I/O (data, file and binlog) except for log I/O; (3) CDBs use shared distributed block storage instead of local storage. Therefore, in contrast to the restarting process of general databases like MySQL where the redo recovery in the storage tier follows the undo operation in compute tier, the decoupled design between compute tier and storage tier in CDBs allows the storage tier to continuously construct the latest version of the data in parallel with the compute tier in an asynchronous manner. Besides, CDBs offload most of the I/O access in this architecture, minimizing the bandwidth between the compute and storage tiers. Thus, CDBs achieve a nearly instant restart, where the restart time is mostly around 5 s. The reduction of the restart time significantly reduces the waiting time (online tuning) for cloud database users of \({\texttt {CDBTune}}^{+}\) and thus provides them with a better experience in practical use (detailed see in Sect. 6.1.3). In addition, if the user is not sensitive to the online tuning time, \({\texttt {CDBTune}}^{+}\) can try more steps in the same online tuning time and might find better configurations to achieve higher performance (detailed see Sect. 6.1.4).

We first evaluate impact of the PER Method in \({\texttt {CDBTune}}^{+}\) and the execution time details of our method, then we compare with baselines and show the performance of varying the tuning steps, neural networks and the types of metrics data.

In this section, we evaluate the effect of varying numbers of knobs and discuss the performance of CDBTune, the DBA, OtterTune and Bestconfig with different workloads. For the database instance CDB-B, we record the throughput, latency, and number of iterations for the TPC-C workload when the model converges. Note that some knobs do not need to be tuned, e.g., those knobs that do not make sense (e.g., path names) to tune or those that are not allowed to tune (which may lead to hidden or serious problems). Such knobs are added to the black-list according to the DBA or user’s demand. We finally operate on 266 tunable knobs (the maximum number of knobs that the DBA uses to tune a CDB). Note that we also provide more experiments on other databases and storage media.

In this section, we first evaluate the benefits of our custom reward function, then explore how the coefficients \(C_{T} \) and \(C_{L}\) will affect the performance of a DBMS.

We evaluate the adaptability of our system, e.g., how our method can adapt itself to a new environment or new workload.