Deep reinforcement learning based on balanced stratified prioritized experience replay for customer credit scoring in peer-to-peer lending

In actual CCS, banks or financial institutions determine whether to grant loans to customers based on customer credit. Even if customers have different credit grades, the final result is still ‘granting’ or ‘not granting’. Therefore, CCS can be regarded as a binary classification problem.

Currently CCS models stemming from operations research and artificial intelligence have also become popular, include LR (Hosmer et al. 2013), NB (Rish 2001), DT (Yeo and Grant 2018), KNN (Wauters and Vanhoucke 2017), SVM (Trafalis and Gilbert 2006), and DNN (Gunnarsson et al. 2021) models. For instance, Lessmann et al. (2015) compared the 41 classification models performance on eight CCS data sets and validated that the DNN model achieved excellent performance. Fernandes and Artes (2016) introduced a measure of the local default risk based on the application of ordinary kriging to logistic credit scoring models. These models achieved better performance on the Brazilian dataset. Li et al. (2020) proposed a recursive Bayes estimator to improve the precision of credit scoring by incorporating the dynamic interaction topology of customers. The experimental results showed that, under the proposed framework, the designed estimator achieved a higher precision than any efficient estimator. Xiao et al. (2021) compared DNN, LDA, LR, DT, and SVM models performance. The experimental results on seven CCS datasets showed that the LR model provided the best performance. Wang et al. (2022) proposed an innovative DQN model for CCS and compared the performance of the proposed DQN model with those of eight other classification models. The experimental results obtained using five CCS datasets showed that the proposed model performed significantly better than the other eight traditional classification models.

In recent years, P2P lending has developed rapidly with the rise of the Internet, and scholars have begun to focus on customer credit scores in P2P lending. For instance, Guo et al. (2016) designed an instance-based credit scoring model that can assess the return and risk of loans. To verify the proposed model, the authors conducted extensive experiments on two actual CCS datasets in P2P lending. The experimental results showed that this model could effectively improve investment performance. Wang et al. (2021) proposed a misclassification cost matrix for P2P credit grading, using a set of equations and models to calculate costs. The results obtained on the Lending Club dataset showed that cost-sensitive classifiers could significantly reduce the total cost. Bastani et al. (2019) proposed a two-stage scoring method based on credit and profit. The first stage identifies the NPLs. The second stage predicts profitability based on the internal rate of return. In both stages, wide and deep learning were used to build the prediction models. The results obtained using the Lending Club dataset showed that the proposed model outperformed the existing credit scoring and profit scoring methods.

In summary, in real-world credit scoring, especially in P2P lending, data are generated in the dynamic interaction between customers and financial institutions, which means that a sequence correlation may exist between the samples, which can affect the CCS performance of the traditional classification model. The DQN-BSPER model proposed in this paper is a sequential decision model and verifies whether there is a sequence correlation between samples in the CCS datasets in P2P lending.

DRL has been widely used in various real-world fields (Moor et al. 2022; Fan et al. 2020; Liu et al. 2020; Patel et al. 2019; Silver et al. 2018), and the most popular model is the DQN model (Mnih et al. 2013, 2015). Over the years, increasingly many scholars have begun using the DQN model for supervised classification. For instance, Zhao et al. (2016) proposed a DQN model for image classification and experimentally proved that the proposed model was highly competitive on the vehicle classification datasets. Lin et al. (2020) proposed a DQN based on an improved reward function (DQN-IRF) model. The model provides more rewards to minority experience samples to make the classification strategy more inclined toward the minority, which effectively improves the classification performance of the model when applied to image and text datasets.

In binary classification, the DQN model have shown excellent performance. For instance, Lopez-Martin et al. (2020) used four DRL models for intrusion detection and verified that their performance were better than traditional classification models. Ding et al. (2019) constructed a DQN model with RER technology to identify machinery running faults and achieved 100% recognition accuracy. Lim et al. (2021) used a DQN model with RER technology to intelligently predict the hidden relationships in criminal network. The experimental results showed that the proposed model performance was superior to RF and SVM. Lin et al. (2020) proposed a DQN model with RER technology for classification task through making the results inclined toward the minority class, and then verify that the improved DQN model was significantly superior to the DNN model on the image and text datasets. In addition, Chen et al. (2018) introduced stratified sampling technology into the experience replay process of the DQN model to improve its convergence performance. Schaul et al. (2015) introduced the degree of importance (referred to as priority) into the experience replay process and developed PER technology. This technology firstly determines the priority of each experience sample according to the TD error. Subsequently, it selects important experience samples to construct the mini-batch based on the priority to optimize the loss function. They proved that the improved process is very effective in improving convergence for DRL models.

In summary, previous scholars have mainly constructed mini-batches based on RER, SER, or PER for DQN models. It is difficult to balance the numbers of minority and majority experience samples in the mini-batch, which may affect the DQN model performance in imbalanced classification. Our proposed BSPER technique fully considers the class-imbalanced characteristics of the CCS datasets in P2P lending and improves the convergence performance of DQN.

For convenience and clarity, the main mathematical notations and definitions used in this article are presented in “Appendix 1”.

Reinforcement learning (RL) is a subclass of machine learning that aims to optimize the action strategy of the agent continuously to maximise the expected cumulative reward in the process of interaction with the environment, that is, to maximise the Q-function (Sutton and Barto 1998). In previous studies, RL tasks have usually been formulated as MDPs (Cai et al. 2020; Zhang et al. 2021), and their basic elements can be expressed as a tuple \(\left({\varvec{S}},{\varvec{A}},P,R,\gamma \right)\), where \({\varvec{S}}\) indicates the state space, \({\varvec{A}}\) indicates the action space, \(P:{\varvec{S}}\times {\varvec{A}}\times {\varvec{S}}\to [\mathrm{0,1}]\) indicates the state transition probability, \(R:{\varvec{S}}\times {\varvec{A}}\to {\mathbb{R}}\) indicates the reward function, and \(\gamma \in [\mathrm{0,1}]\) indicates the discount factor that balances the importance of future rewards and current rewards. In particular, during the tth (\(t\in [0,T]\)) time step, the agent first performs an action \({a}_{t}\in {\varvec{A}}\) under the environment state \({s}_{t}\in {\varvec{S}}\). Then, the reward \({r}_{t}\) is generated by environment and feeds it back to the agent. Finally, the environment is transferred to the next state \({s}_{t+1}\) according to probability \(P\). The cumulative reward from \({s}_{t}\) to \({s}_{T}\) can be expressed as \({R}_{t}={\sum }_{i=t}^{T}{\gamma }^{i-t}{r}_{i}\). In addition, the expected cumulative reward (usually represented by the Q-function) corresponding to the state–action pair \(\left({s}_{t},{a}_{t}\right)\) is expressed as \(Q\left({s}_{t},{a}_{t}\right)=E[{\sum }_{i=t}^{T}{\gamma }^{i-t}{r}_{i}]\) (Chen et al. 2018), and the optimal strategy \({\pi }^{*}\) represents the strategy that can maximise the Q-function. According to the Bellman equation (Gosavi 2009), the Q-function can be transformed into the following form:where \(\underset{{a}_{t+1}\in {\varvec{A}}}{{\text{max}}}Q\left({s}_{t+1},{a}_{t+1}\right)\) represents the maximum Q-value that the agent can obtain when the state is \({s}_{t+1}\).

Q-learning is a widely used model-free RL algorithm based on asynchronous dynamic programming that can quickly find the optimal strategy for the MDP (Watkins and Dayan 1992). The core idea is to find the optimal strategy \({\pi }^{*}\) that can maximise the Q-value using the Bellman equation (Gosavi 2009) to iterate the Q-table continuously. The general steps of Q-learning can be summarised as follows:

Step 2: According to the Q-table, the agent executes action \({a}_{t}\) in state \({s}_{t}\).

Step 3: When the state-action pair is \(\left({s}_{t},{a}_{t}\right)\), the agent obtains reward \({r}_{t}\) according to the reward function. Simultaneously, the next state \({s}_{t+1}\) is generated from environment, then iteratively updates the Q-value using the Bellman equation (Gosavi 2009):where \(\alpha\) represents the learning rate.

Step 4: Repeat Steps 2 and 3 until none of the Q-values in the Q-table change. Then, the strategy corresponding to the Q-table is the optimal strategy \({\pi }^{*}\) (Sutton and Barto 1998).

Q-learning algorithms have been successfully applied in many fields of real word, but they are primarily suitable for tasks with small state spaces. In the real world, the state spaces of tasks are typically large, and the number of states can even reach tens of millions. To solve this problem, Mnih et al. (2015) combined a DNN with Q-learning to develop a DQN model, which approximately expresses the Q-function by automatically extracting features from the state space. Specifically, experience samples are continuously obtained first according to the greedy strategy, and they are stored in a fixed-size replay memory buffer to form an experience sample set. In particular, the experience sample at the tth time step is represented as \({e}_{t}=({s}_{t},{a}_{t},{r}_{t},{s}_{t+1})\), and the experience sample set is represented as \({{\varvec{E}}}_{t}=\{{e}_{1},{e}_{2},\dots ,{e}_{t}\}\). Then, \(k\) experience samples are randomly selected from \({{\varvec{E}}}_{t}\) to form a mini-batch using RER technology. Finally, a DNN is used to fit the Q-function, so the network corresponding to the Q-function is called the Q-network, and its general expression is \(Q\left(s,a;\theta \right)\), whereas the loss function of \(L({\theta }_{t}^{C})\) can be expressed aswhere \({{y}_{i}=r}_{i}+\gamma \underset{{a}_{i+1}\in {\varvec{A}}}{{\text{max}}}Q\left({s}_{i+1},{a}_{i+1};{\theta }_{t}^{T}\right)\) indicates the target Q-function; \({\theta }_{t}^{T}\) indicates the parameters of the target Q-network; \(Q\left({s}_{i},{a}_{i};{\theta }_{t}^{C}\right)\) indicates the current Q-function; \({\theta }_{t}^{C}\) indicates the parameters of the current Q-network. The gradient descent algorithm was used to update the parameters of the Q-network:

where \(\alpha\) indicates the learning rate of the Q-network. After training, we can obtain the optimal strategy \({\pi }^{*}=\underset{a\in {\varvec{A}}}{argmax}{Q}^{*}\left(s,a;\theta \right)\), which can maximise the Q-function. In addition, iterative updating technology is used to reduce the correlation between the current Q-network and target Q-network. In other words, the parameter of the current Q-network \({\theta }_{t}^{C}\) is assigned to the parameter of the target Q-network \({\theta }_{t}^{T}\) after a certain number of steps for improving the convergence stability of the Q-network (Lopez-Martin et al. 2020; Mnih et al. 2015). For the detailed algorithm, see the literature (Mnih et al. 2015).

In most DRL frameworks, the agent constantly receives new experience samples to update the parameters of the DQN model incrementally. The simplest update method uses only one experience sample in each time step to update the model parameters. However, the biggest drawback of this method is that the rare experiences that may be useful in the future will be quickly forgotten, resulting in low sampling efficiency. Experience replay can effectively solve this issue (Luo et al. 2018). In other words, experience samples are firstly stored in a fixed-size replay memory buffer; then, each time the parameters of the DQN model are updated, a fixed number of experience samples are sampled from the replay memory buffer to construct a mini-batch; and finally, the gradient descent algorithm is used to train and optimize the model.

Obviously, a complete experience replay process consists of storing and sampling the experience samples. Therefore, the selection of experience samples from the replay memory buffer is important in improving the DQN model performance. In particular, when an experience sample is stored in replay memory buffer \(D\), a new index label \(i\in \left\{\mathrm{1,2},\dots ,d\right\}\) is assigned to the experience. The priority of the ith experience sample is represented as \(P({e}_{i})\). The entire replay memory buffer can be regarded as a combination of experience samples and priority \(\left\{{e}_{i},P({e}_{i})\right\}\). The key to selecting the experience samples is to determine \(P({e}_{i})\) for each experience sample. The most commonly used experience replay technology in the DQN model is RER, and the priority of each experience sample is the same, that is, \(P\left({e}_{i}\right)=\frac{1}{d}\). This technology uses a random sampling method to select experience samples from the buffer to construct the mini-batch, which ignores the experience samples that play an important role in the parameter updating of the DQN model. Consequently, the DQN model may converge slowly in complex tasks. To solve this problem, Schaul et al. (2015) proposed PER technology. The core steps of this technology can be summarised as follows. Firstly, the TD error of the experience sample in the replay memory buffer can be calculated as follows:

The priority of each experience sample is then determined according to the TD error; that is, the selected probability of the experience sample can be expressed as

Finally, \(k\) experience samples are selected from the replay memory buffer according to the probability \(P\left({e}_{i}\right)\) to construct a mini-batch. The higher the TD error, the higher the priority of the experience sample. Thus, a sample with a higher TD error is selected with a higher probability, which effectively improves the convergence performance of the DQN model (Schaul et al. 2015).

Furthermore, Chen et al. (2018) developed an SER by introducing stratified sampling technology into the experience replay for the DQN model. The core idea of this technology is to use a stratified sampling method to select different classes of experience samples from the replay memory buffer to construct a mini-batch. The experimental results demonstrate that this technology can dramatically improve the classification performance of the DQN model.

The most commonly used experience replay technology in the DQN model is RER, which randomly selects experience samples from the buffer to construct a mini-batch. The main advantage of this method is that it is easy to implement. However, the mini-batch constructed by RER technology has an imbalanced class distribution in CCS, and it is difficult to select experience samples that play important roles in updating the loss function parameters, which may affect the CCS performance of the DQN model. SER can reduce the effects of the imbalanced class distribution on the DQN model performance by adjusting the numbers of minority and majority experience samples in the mini-batch (Chen et al. 2018). However, it is difficult to select experience samples that play important roles in updating the loss function parameters. To avoid the shortcomings of the SER, PER provides a concept that can select more important experience samples to improve the convergence performance of the DQN model (Schaul et al. 2015), but the class distribution of the constructed mini-batch is still imbalanced. It can be observed that SER and PER are complementary in the experience replay process. Therefore, we combined SER and PER to develop the BSPER technology.

To express the sampling process of the BSPER technology more clearly, we provide definitions of the majority and minority experience samples. If the current state of an experience sample is a minority sample (a positive class sample), then the sample is the minority experience sample, which is represented as \({{\varvec{E}}}^{min}\), and the tth minority experience sample is represented as \({e}_{t}^{min}=({s}_{t}^{min},{a}_{t}^{min},{r}_{t}^{min},{s}_{t+1}^{min})\). If the current state of an experience sample is a majority class sample (a negative class sample), then let the sample be the majority experience sample, which is represented as \({{\varvec{E}}}^{maj}\), and let the tth majority experience sample be indicated as \({e}_{t}^{maj}=({s}_{t}^{maj},{a}_{t}^{maj},{r}_{t}^{maj},{s}_{t+1}^{maj})\), \({{\varvec{E}}}^{min}\cap {{\varvec{E}}}^{maj}=\varnothing\). The core steps of the SPER technology can be described as follows.

Step 1: Store the experience samples in the minority and majority experience replay buffers \({D}^{min}\) and \({D}^{maj}\), respectively, according to the form of SumTree (Schaul et al. 2015), and obtain the replay memory sets of the minority and majority experience samples \({{\varvec{E}}}_{replay}^{min}={\left\{{e}_{i}^{min}\right\}}_{1\le i\le {d}_{1}}\subset {{\varvec{E}}}^{min}\) and \({{\varvec{E}}}_{replay}^{maj}={\left\{{e}_{j}^{maj}\right\}}_{1\le j\le {d}_{2}}\subset {{\varvec{E}}}^{maj}\).

Step 2: Calculate the TD errors of the experience samples in \({{\varvec{E}}}_{replay}^{min}\) and \({{\varvec{E}}}_{replay}^{maj}\), denoted as \({\delta }_{i}^{min}\) and \({\delta }_{j}^{maj}\), respectively, which can be expressed as follows:

where \({y}_{i}^{min}={r}_{i}^{min}+\gamma \underset{{a}_{i+1}^{\mathit{min}}\in {\varvec{A}}}{\cdot {\text{max}}}Q\left({s}_{i+1}^{min},{a}_{i+1}^{min};{\theta }^{T}\right)\) and \({y}_{j}^{maj}={r}_{j}^{maj}+\gamma \underset{{a}_{j+1}^{\mathit{maj}}\in {\varvec{A}}}{\cdot {\text{max}}}Q\left({s}_{j+1}^{maj},{a}_{j+1}^{maj};{\theta }^{T}\right)\) indicate the target Q-functions in the minority and majority experience samples, respectively; \(Q\left({s}_{i}^{min},{a}_{i}^{min};{\theta }^{C}\right)\) and \(Q\left({s}_{j}^{maj},{a}_{j}^{maj};{\theta }^{C}\right)\) indicate the current Q-functions in the minority and majority experience samples, respectively; \({\theta }^{T}\) and \({\theta }^{C}\) indicate the parameters of the target Q-network and current Q-network, respectively; and \({d}_{1}\) and \({d}_{2}\) indicate the numbers of minority and majority experience samples in \({{\varvec{E}}}_{replay}^{min}\) and \({{\varvec{E}}}_{replay}^{maj}\), respectively.

Step 3: Calculate the probabilities of minority and majority experience samples selected from \({{\varvec{E}}}_{replay}^{min}\) and \({{\varvec{E}}}_{replay}^{maj}\) according to the TD error, which are represented as \(P\left({e}_{i}^{min}\right)\) and \(P\left({e}_{j}^{maj}\right)\), respectively:

Step 4: Select \({k}_{1}\) and \({k}_{2}\) (\({k}_{1}={k}_{2}\)) minority and majority experience samples from \({{\varvec{E}}}_{replay}^{min}\) and \({{\varvec{E}}}_{replay}^{maj}\) according to \(P\left({e}_{i}^{min}\right)\) and \(P\left({e}_{j}^{maj}\right)\), respectively, to construct the stratified prioritized mini-batch.

This section describes the introduction of the BSPER technology into the DQN model to construct the DQN-BSPER model. Figure 1 shows the Q-network structure of the DQN-BSPER model. The input layer of the network is the environmental state (the features of the customer sample) \({s}_{t}=\left({s}_{t}^{1},{s}_{t}^{2},\dots {,s}_{t}^{g}\right)\), where the number of nodes in the input layer is the feature number of environment state \(g\), and the output layer is set to two Q-values. According to the Q-value, the customer class label is mapped using a one-hot method, and the nodes in each layer are connected in a fully connected manner.

The core idea of the DQN-BSPER model for CCS is as follows. Firstly, in the training set, the BSPER technology is used to construct the stratified prioritized mini-batch to optimize the loss function, and then the gradient descent algorithm is used to update the Q-network parameters continuously to optimize the classification strategy of the agent in CCS. Finally, in the test set, the trained DQN-BSPER model was used to evaluate customer credit. The detailed modelling steps of the proposed model can be summarised as follows (the modelling process is shown in “Appendix 2”).

Step 1: Initialise the parameters of the current Q-network \({\theta }_{t}^{C}\), parameters of the target Q-network \({\theta }_{t}^{T}\), target network update frequency \(Z\), time step \(t\), maximum time step \(T\), episode \(c\), maximum episode \(C\), size of mini-batch \(k\), number of minority experience samples in mini-batch \({k}_{1}\), and number of majority experience samples in mini-batch \({k}_{2}\).

Step 2: Randomly sort the customer samples in the training set and input them into the CCS environment. Each environmental state corresponds to a customer sample; that is, \({s}_{t}\leftarrow {x}_{t}\).

Step 3: The agent obtains environment state \({s}_{t}\) and performs credit scoring actions \({a}_{t}\) according to the ε-greedy strategy:where the ε-greedy strategy adds randomness to the greedy strategy. That is, the agent selects the action that maximises \(Q\left({s}_{t},{a}_{t};{\theta }_{t}^{C}\right)\) based on the probability \(1-\varepsilon\) under environment state \({s}_{t}\); otherwise, an action is randomly selected based on the probability \(\varepsilon\). Then, reward \({r}_{t}\) is obtained according to the reward function \(R\left({a}_{t},{y}_{t}\right)\) (see Eq. (8)), and the environment is transferred to the next state \({s}_{t+1}\).

Step 4: According to Step 3, the agent continuously obtains the experience sample \({e}_{t}=\left({s}_{t},{a}_{t},{r}_{t},{s}_{t+1}\right)\), and obtains \({{\varvec{E}}}_{replay}^{min}\) and \({{\varvec{E}}}_{replay}^{maj}\) by using the BSPER technology.

Step 5: If \(\left|{{\varvec{E}}}_{replay}^{min}\right|\le {k}_{1}\) or \(\left|{{\varvec{E}}}_{replay}^{maj}\right|\le {k}_{2}\), then go to Step 3 and let \(t\leftarrow t+1\); otherwise, use the BSPER technology to construct the stratified prioritized mini-batch. Then, the corresponding loss function of Q-network at the t-th time step \({L}_{STS}\left({\theta }_{t}^{C}\right)\) can be expressed as follows: where \({y}_{i}^{min}\) and \({y}_{j}^{maj}\) indicate the target Q-functions in the minority and majority experience samples, respectively; \(Q\left({s}_{i}^{min},{a}_{i}^{min};{\theta }_{t}^{C}\right)\) and \(Q\left({s}_{j}^{maj},{a}_{j}^{maj};{\theta }_{t}^{C}\right)\) indicate the current Q-functions in the minority and majority experience samples, respectively; \({\theta }_{t}^{T}\) and \({\theta }_{t}^{C}\) indicate the parameters of the target and current Q-networks, respectively; \({k}_{1}\) and \({k}_{2}\) (\({k}_{1}+{k}_{2}=k\)) indicate the numbers of minority and majority experience samples selected from \({{\varvec{E}}}_{replay}^{min}\) and \({{\varvec{E}}}_{replay}^{maj}\), respectively; and \(\gamma\) is the discount factor.

Step 6: Use the Adam optimization algorithm to update the parameters of the current Q-network (Schaul et al. 2015): where \(\frac{\nabla {L}_{STS}\left({\theta }_{t}^{C}\right)}{\nabla {\theta }_{t}^{C}}\) indicates the gradient of \({L}_{STS}\left({\theta }_{t}^{C}\right)\) at the tth time step; \({m}_{t}\) and \({d}_{t}\) indicate the first- and second-order moments of \(\frac{\nabla {L}_{STS}\left({\theta }_{t}^{C}\right)}{\nabla {\theta }_{t}^{C}}\), respectively; \({\beta }_{1}\) and \({\beta }_{2}\) indicate the exponential decay rates of the first- and second-order moments, respectively; \({\widehat{m}}_{t}\) and \({\widehat{d}}_{t}\) indicate the deviation correction values of \({m}_{t}\) and \({d}_{t}\), respectively; Eq. (24) indicates that the parameters of the Q-network \({\theta }_{t}^{C}\) are updated by combining Eqs. (22) and (23); \(\alpha\) indicates the learning rate of the current Q-network; and \(\epsilon\) indicates a constant value to prevent \(\sqrt{{\widehat{d}}_{t}}+\epsilon\) from generating the 0 value. Then, for each \(Z\) time step, the parameters of the current Q-network \({\theta }_{t}^{C}\) are assigned to the parameters of the target Q-network \({\theta }_{t}^{T}\); that is, let \({\theta }_{Z|t}^{T}\leftarrow {\theta }_{Z|t}^{C}\).

Step 7: If \(t\le T\), then proceed to Step 3 and let \(t\leftarrow t+1\); otherwise, proceed to Step 8.

Step 8: If \(c\le C\), then go to Step 2 and let \(c\leftarrow c+1\); otherwise, stop training, output the trained current Q-network, and use it to conduct CCS according to the greedy strategy in the test set.

The current Q-network obtained through the above steps is a parameterised optimal Q-function; therefore, the strategy corresponding to the network is also the optimal classification strategy. The pseudo-codes of the CCS environment simulation and DQN-BSPER model are shown in “Appendix 3”.

To analyse the CCS performance of the DQN-BSPER model, we selected two CCS datasets in P2P lending (IFCD and Leading Club). The IFCD dataset was obtained from an Internet financial company in China. The original dataset contained 1110 features. The Leading Club dataset was acquired from the first quarter data of Lending Club, which is a P2P lending platform in the United States in 2016, and the original dataset contained 110 features. According to the regulations of the Basel Banking Regulatory Association, we labelled customers whose loans were overdue for more than 90 days as having bad credit and others as having good credit.

The two CCS datasets in P2P lending contained redundant features and missing values, which were processed as follows. First, we deleted some meaningless features through observation, such as loan number, zip code, and customer ID. Then, the features with missing rates of more than 30% were eliminated. Finally, the recursive feature elimination method Wrapper was used to eliminate the features with the minimum absolute weights continuously. The basic information about the two CCS datasets in P2P lending after preprocessing is shown in Table 1, including the dataset name (the abbreviation in parentheses), number of features, number of customer samples, number of customer samples with good credit, number of customer samples with bad credit, and imbalanced ratio (IR), which is defined as the proportion of the number of the majority samples (customers with good credit) relative to the number of minority samples (customers with bad credit). Obviously, the greater the \(IR\), the higher the class distribution imbalance. As shown in Table 1, the two CCS datasets are imbalanced.

In this study, we performed fivefold cross validation in each dataset to obtain the calculation results for the models. Firstly, each dataset was divided equally into five subsets at random. In each experiment, one subset was used as the test set, \({{\varvec{D}}}_{test}\), and the remaining subsets were used as the training set, \({{\varvec{D}}}_{train}\). Then, the DQN-BSPER model and other models were trained in \({{\varvec{D}}}_{train}\). Finally, the trained models were used to classify the samples in \({{\varvec{D}}}_{test}\). This process was repeated five times to ensure that each subset was used once. The entire process constitutes fivefold cross validation. To obtain more stable and reliable experimental results, we repeated the fivefold cross validation 10 times and took the average value as the final calculation result. In addition, we recorded the network loss each time the model traversed the training set in the process of fivefold cross validation and averaged the training network loss at the end of training. All experiments in this study were run on Python 3.6, in a Windows 10 × 64 bit system equipped with an Intel (R) Core (TM) i9 processor.

We mainly referred to previous studies (Lin et al. 2020; Liu et al. 2020; Mnih et al. 2015) to set the parameters of the DRL models. “Appendix 4” shows the main parameter settings of the DQN-BSPER and other DRL models. In particular, the size of the replay memory buffers of the DQN-BSPER and DQN-SER models \({D}^{min}\) and \({D}^{maj}\) was 500. In addition, the current and target Q-networks of the DQN-BSPER model had the same structure. To ensure the fairness of the experiment, we adjusted the discount factors of the DQN, DQN-SER, DQN-PER, and DQN-IRF models many times and determined their optimal values. Specifically, the optimal discount factor of the DQN-SER model was 0.1, and the optimal discount factor of the DQN, DQN-PER, and DQN-IRF models was 0.3. In all experiments, the network parameters of the five DRL models were equal. In addition, we used the grid search and fivefold cross validation (Gunnarsson et al. 2021) to ensure the excellent performance of the other classification models.

The confusion matrix method can intuitively evaluate classification models performance (Batista et al. 2004). Table 2 shows the confusion matrix of CCS, from which many different EMs can be obtained.

However, there are some contradictions between these EMs, so some new comprehensive measures have been proposed, such as F1 (Tang et al. 2008) and AUC (Bradley 1997). F1 is a combination of precision and recall:where recall = TPR and \(\beta\) is the relative coefficient of recall to precision. In this study, we design \(\beta =1\).

In this research, we used the approximate representation of the AUC for the binary classification problem (Loyola-González et al. 2016):

Therefore, we used four TPR, ACC, F1, and AUC in this study. The larger the values of them, the better the performance of the CCS model.

To analyse the effects of the discount factor \(\gamma\) and the proportion of minority and majority experience samples in the stratified prioritized mini-batch \({k}_{1}/{k}_{2}\) on the CCS performance of the DQN-BSPER model, we set 11 different types of\(\gamma\); that is, we let\(\gamma =\{\mathrm{0,0.1,0.2},\dots ,1\}\). A large difference between \({k}_{1}\) and \({k}_{2}\) may significantly affect the DQN-BSPER model performance. Therefore, we set 11 different values of\({k}_{1}/{k}_{2}\): {\(6/1 , 5/1, 4/1, \dots , 1/6\}\). Owing to the lack of prior experience, we temporarily set \({k}_{1}/{k}_{2}=1\) when analysing the effects of the discount factor on the DQN-BSPER model performance.

Figure 2 depicts the average values of the EMs of the DQN-BSPER model under 11 values of \(\gamma\) (the average value of each EM is the average value of the classification results of the model on two CCS datasets in P2P lending). “Appendix 5” shows the CCS performance of the DQN-BSPER model under the 11 discount factors. The bold values indicate the best CCS performance in each column, the numbers in parentheses indicate the rankings of the DQN-BSPER model with different discount factors, and the last column indicates the average ranking value of the DQN-BSPER model performance with each discount factor. The smaller the ranking value, the better the model performance. As shown in Fig. 2, with respect to TPR, the CCS performance of the DQN-BSPER model shows a gradual upward trend with increasing \(\gamma\). According to ACC, F1, and AUC, the DQN-BSPER model performance shows a gradual downward trend with increasing \(\gamma\). To determine the most appropriate discount factor, we ranked the DQN-BSPER model performance under different discount factors. As can be seen in “Appendix 5”, the DQN-BSPER model has the smallest ranking value when \(\gamma =0.1\). The DQN-BSPER model has the largest ranking value when \(\gamma =0.8\).

Figure 3 presents the average values of the EMs of the DQN-BSPER model under 11 values of \({k}_{1}/{k}_{2}\). Furthermore, we ranked the CCS performance of the DQN-BSPER model under different \({k}_{1}/{k}_{2}\) values, and the results are shown in “Appendix 6”. As depicted in Fig. 3, with respect to TPR, the DQN-BSPER model performance shows a gradual downward trend with increasing \({k}_{1}/{k}_{2}\). With respect to ACC, the DQN-BSPER model performance shows a gradual upward trend with increasing \({k}_{1}/{k}_{2}\). With respect to F1 and AUC, the DQN-BSPER model performance firstly shows an upward trend and then a downward trend. As shown in “Appendix 6”, the DQN-BSPER model has the smallest ranking value when \({k}_{1}/{k}_{2}=1\).

It can be seen from the experimental results that the CCS performance of the DQN-BSPER model is the best when \(\gamma =0.1\) and \({k}_{1}/{k}_{2}=1\). Interestingly, with increasing \(\gamma\), the TPR gradually increases, whereas ACC, F1, and AUC gradually decrease. This observation shows that the greater \(\gamma\), the more majority samples will be wrongly divided into the minority by the DQN-BSPER model, resulting in poor overall performance of the model. When \(\gamma =0.1\), the DQN-BSPER model performance is the best, which also demonstrates that there is a weak sequence correlation between the samples in the CCS dataset, whereas identifying the correlation may make the DQN-BSPER model more robust (Lei et al. 2020). In addition, setting \({k}_{1}/{k}_{2}\) too high or too low leads to poor CCS performance of the DQN-BSPER model. When \({k}_{1}/{k}_{2}=1\), the DQN-BSPER model has the highest F1 and ACC values, which verifies that balancing the class distribution of different experience samples in the stratified prioritized mini-batch can improve the CCS performance of the DQN-BSPER model the most effectively.

This section compares the CCS performance of DQN-BSPER and the other four benchmark DRL models, DQN, DQN-SER, DQN-PER, and DQN-IRF, on the IF and LE datasets according to four EMs. Table 3 presents the CCS performance of DQN-BSPER and the other four benchmark DRL models on the IF and LE datasets. The bold font indicates the model with the best CCS performance in each column, and the numbers in parentheses indicate the model performance rankings. The last column indicates the average ranking of the performance of each model.

Table 3 demonstrates that the DQN-BSPER model has the smallest average ranking, indicating that the model has the best CCS performance. To analyse whether there are statistically significant differences in CCS performance among DQN-BSPER and the other four benchmark DRL models, we used the nonparametric Wilcoxon rank sum test (Wilcoxon 1992). The null hypothesis is that the two comparative models have the same CCS performance. In this paper, we set the significance level \(\alpha =0.05\). Then, at the 95% confidence level, when the statistical amount is 8 (\(=2\times 4\)), the critical value (CV) is 3. The Wilcoxon rank sum test results of DQN-BSPER and four other benchmark models are provided in Table 4. In Table 4, if T = Min(R⁺, R⁻) \(\le\) 3, the null hypothesis is rejected. In particular, if T = R⁻ and T \(\le\) 3, it means that the former performance is statistically significantly better than the latter. If T = R⁺ and T \(\le\) 3, then the opposite is true.

The test results in Table 4 indicate that when comparing the DQN-BSPER model with the DQN, DQN-SER, DQN-PER, and DQN-IRF models, R⁺ is less than the CV; in other words, at the 95% confidence level, the CCS performance of the DQN-BSPER model is statistically significantly better than those of the other four benchmark models. More importantly, the CCS performances of the DQN-SER and DQN-PER models are better than that of the DQN model, whereas the DQN-BSPER model is superior to the DQN-SER and DQN-PER models. Thus, although the CCS performance of the DQN model can be improved by using SER or PER technology alone, the effect of improvement is very limited, whereas the combination of SER and PER technologies can further improve the performance of the DQN model. In addition, for the IF dataset, the CCS performance of the DQN-IRF model is very poor. This poor performance may occur because the DQN-IRF model (Lin et al. 2020) is mainly applied to classification tasks with unstructured data (image and text data), whereas the CCS data are structured, which may lead to the failure of the reward function, affecting the CCS performance of the DQN-IRF model.

To analyse the effects of the SPER technology on the convergence performance of the DQN model, this section compares the convergence performances of the DQN-BSPER, DQN, DQN-SER, DQN-PER, and DQN-IRF models according to four EMs on the IF and LE datasets. Figure 4 shows the average training loss of the Q-networks of DQN-BSPER and the other four benchmark DRL models on two CCS training sets.

Figure 4a shows the average training losses of the Q-networks of the five DQN models on the IF dataset. The average training loss of the Q-network of the DQN-BSPER model reaches a stable value when episode = 1, indicating that the DQN-BSPER model converges when episode = 1, and the value fluctuates slightly as the number of episodes increases. The average training loss of the Q-network of the DQN model is very large when episode = 1; then, the value decreases rapidly and reaches the local minimum when episode = 4, but the value fluctuates strongly as the number of episodes increases. The average training loss of the Q-network of the DQN-SER model is larger than those of the other models, and the fluctuation is also very strong. The average training losses of the Q-networks of DQN-PER and DQN-IRF models reach stable values when episode = 3; that is, the DQN-PER and DQN-IRF models reach a convergence state when episode = 3. Figure 4b shows the average training losses of the Q-networks of the five DQN models on the LE dataset. The average training loss of the Q-network of the DQN-BSPER model reaches a stable value when episode = 1, indicating that the DQN-BSPER model converges when episode = 1, and the value fluctuates slightly as the number of episodes increases. The average training loss of the Q-network of the DQN model is very large when episode = 1 and reaches the local minimum when episode = 9, but the value still fluctuates strongly as the number of episodes increases. The average training loss of the Q-network of the DQN-SER model also fluctuates significantly. The average training losses of the Q-networks of the DQN-PER and DQN-IRF models reach stable values when episode = 6 and episode = 3, respectively; that is, the DQN-PER and DQN-IRF models reach a convergence state when episode = 6 and episode = 3, respectively, and their average training losses of the Q-network are relatively small.

The experimental results show that the Q-network of the DQN-BSPER model has the fastest convergence speed and that the average training loss of the Q-network is the most stable. Thus, the SPER technology can effectively improve the convergence performance of the DQN model; that is, it can not only accelerate the convergence speed of the Q-network, but also make the Q-network more stable. In addition, the average training losses of the Q-networks of the DQN-PER and DQN-IRF models are very small, but the results in Sect. 7.2 show that their CCS performance is relatively poor, which indicates that over-fitting may occur. Interestingly, the fluctuation in the average training losses of the DQN and DQN-SER models is stronger than that of the DQN-BSPER and DQN-PER models. The main reason may be that the DQN-BSPER and DQN-PER models consider the priority of the experience samples when building the mini-batch. Thus, PER technology can improve the stability of the DQN model more effectively than SER technology.

This section compares the CCS performance of DQN-BSPER and seven traditional benchmark classification models on the IF and LE datasets. The results are presented in “Appendix 7”, in which the bold font indicates the classification model with the best CCS performance in each line, and the numbers in parentheses indicate the performance rankings of the classification models. The last line indicates the average ranking of each classification model. To analyse whether there are statistically significant differences between DQN-BSPER and the seven traditional benchmark classification models, we used the nonparametric statistical test method proposed by (Demšar 2006), namely, the Friedman test (Friedman 1940) and Iman–Davenport test (Iman and Davenport 1980). If there were statistically significant differences, we further used the Nemenyi post hoc test method to compare the eight classification models. In addition, in order to achieve fair results, we used the resampling methods ROS and RUS to balance the class distribution of the two training sets to form four new training sets (the class distribution ratio of the balanced training set was 1:1) during each training process. Each dataset corresponded to two new cases; that is, the IF dataset corresponded to IF (ROS) and IF (RUS), whereas the LE dataset corresponded to LE (ROS) and LE (RUS).

The results in “Appendix 7” show that the DQN-BSPER model has the smallest average ranking, indicating that it achieves the best CCS performance. We used the Friedman and Iman–Davenport tests to analyse further whether there were statistically significant differences between DQN-BSPER and the seven traditional benchmark classification models. The null hypothesis of the two test methods was that the CCS performance of the eight classification models were the same. We used the \({\chi }^{2}\) distribution with 7 freedom degree and the F distribution with 7 and 161 (\(=7\times 23\)) freedom degree. Table 5 list the test results. If the test value was greater than the distribution value, the null hypothesis was rejected. Therefore, according to results of Table 5, we rejected the null hypothesis and concluded that there were statistically significant differences among the eight classification models in the CCS performance at the 95% confidence level.

After the null hypothesis was rejected, we used the Nemenyi post hoc test. The judgement rule of this test is that if the difference between the average values of any two classification models is greater than the critical difference (CD), then the null hypothesis that the two classification models performance is the same at the 95% confidence level is that there are significant differences in the CCS performance between the two classification models. When the number of classification models is 8 and the CV (Demšar 2006) is 3.03, the corresponding critical distance \({\text{CD}}=3.03\sqrt{(8*9)/(6*24)}\approx 2.14\). The test results are presented in Fig. 5. If there are segments connected between the two classification models, then there are no statistically significant differences. The results in Fig. 5 show that the CCS performance of the DQN-BSPER model is statistically significantly better than those of the other seven traditional benchmark classification models at the 95% confidence level.

To analyse the effects of imbalanced class distribution on the CCS performance of the DQN-BSPER model, we set 10 imbalance ratios for the IF and LE datasets. Specifically, we firstly randomly sampled from the majority samples in the training set according to the multiple of the number of minority samples and then combined the sampled majority samples with all minority samples to form a new training set. According to this method, training sets with 10 imbalance ratios \((IR=\left\{1, 2, \dots , 10\right\})\) were obtained, and the DQN-BSPER model performance was analysed according to four EMs. Figure 6 shows the CCS performance of the DQN-BSPER model under 10 imbalanced ratios on the IF and LE datasets. Furthermore, we used boxplots to analyse the dispersion degrees of the four EM values, as shown in Fig. 7.

As can be seen from Fig. 6a, under 10 types of IR, the values of TPR, ACC, and AUC of the DQN-BSPER model are higher than the F1 values. In Fig. 6b, under 10 types of IR, the values of TPR, ACC, and AUC of the DQN-BSPER model are also higher than the F1 values. When \(IR=1\), the four evaluation values of the DQN-BSPER model are relatively low. In Fig. 7a, the box corresponding to the TPR is the longest, whereas the ACC, AUC, and F1 boxes are relatively short. In Fig. 7b, the box for F1 is the shortest, whereas the TPR, ACC, and AUC boxes are relatively long. It can be seen from the experimental results that for the IF and LE datasets, the boxes of the four EM values of the DQN-BSPER model are very narrow, which demonstrates that their dispersion degree is quite small. Thus, the fluctuations in the TPR, ACC, F1, and AUC values of the DQN-BSPER model are very weak under different imbalance ratios. This finding indicates that the imbalanced class distribution has a limited effect on the CCS performance of the DQN-BSPER model.