Sharpening the Accuracy of Credit Scoring Models with Machine Learning Algorithms

Because of its simplicity, LR is still one of the most popular approaches used in the industry for the classification of applicants (see, e.g., [23]). This approach allows one to model the posterior probabilities of K different applicant classes using a linear function of the features, while at the same time ensuring that the probabilities sum to one and that their value ranges between zero and one. More specifically, when there are only two classes (coded via y, a dummy variable that takes a value of 0 if the applicant is “good” and of 1 if she is “bad”), the posterior probabilities are modeled as Applying the logit transformation, one obtains the log of the probability odds (the log-odds ratio) as The input space is optimally divided by the set of points for which the log-odds ratio is zero, meaning that the posterior probability of being in one class or in the other is the same. Therefore, the decision boundary is the hyperplane defined by \(\left \{x|\beta _0+\vec {\beta }^T\vec {x}=0\right \}\). Logistic regression models are usually estimated by maximum likelihood, assuming that all the observations in the sample are independently Bernoulli distributed, such that the log-likelihood functions is where T ₀ are the observations in the training sample, θ is the vector of parameters, and \(p_k(\vec {x}_i;\theta )=p(C=k|X=\vec {x}_i;\theta )\). Because in our case there are only two classes coded via a binary response variable y _i that can take a value of either zero or one, \(\hat {\beta }\) is found by maximizing

A second popular approach used to separate “good” and “bad” applicants that lead to linear decision boundaries is LDA. The LDA method approaches the problem of separating two classes based on a set of observed characteristics \(\vec {x}\) by modeling the class densities \(f_G(\vec {x})\) and \(f_B(\vec {x})\) as multivariate normal distributions with means \(\vec {\mu }_G,\) and \(\vec {\mu }_B\) and the same covariance matrix Σ, i.e., To compare the two classes (“good” and “bad” applicants), one has then to compute and investigate the log-ratio which is linear in \(\vec {x}\). Therefore, the decision boundary, which is the set where p(C = G|X = x) = p(C = B|X = x), is also linear in \(\vec {x}\). Clearly the Gaussian parameters \(\vec {\mu }_G\), \(\vec {\mu }_B\), and Σ are not known and should be estimated using the training sample as well as the prior probabilities π _G and π _B (set to be equal to the proportions of good and bad applicants in the training sample). Rearranging Eq. (10), it appears evident that the Bayesian optimal solution is to predict a point to belong to the “bad” class if which can be rewritten as where \( \vec {w}=\hat {\varSigma }^{-1}(\hat {\vec {\mu }}_B-\hat {\vec {\mu }}_G)\) and \(z=\frac {1}{2}\hat {\vec {\mu }}_B^T\hat {\varSigma }^{-1}\hat {\vec {\mu }}_B-\frac {1}{2}\hat {\vec {\mu }}_G^T\hat {\varSigma }^{-1}\hat {\vec {\mu }}_G+\log {\hat {\pi }}_G - log{\hat {\pi }}_G\).

Another way to approach the problem, which leads to the same coefficients \(\vec {w}\) is to look for the linear combination of the features that gives the maximum separation between the means of the classes and the minimum variation within the classes, which is equivalent to maximizing the separating distance M Notably, the derivation of the coefficients \(\vec {w}\) does not require that \(f_G(\vec {x})\) and \(f_B(\vec {x})\) follow a multivariate normal as postulated in Eq. (9), but only that Σ _G = Σ _B = Σ. However, the choice of z as a cut-off point in Eq. (12) requires normality. An alternative is to use a cut-off point that minimizes the training error for a given dataset.

The Naïve Bayes (NB) approach is a probabilistic classifier that assumes that given a class (G or B), the applicant’s attributes are independent. Let π _G denote the prior probability that an applicant is “good” and π _B the prior probability that an applicant is “bad.” Then, because of the assumption that each attribute x _i is conditionally independent from any other attribute x _j for i ≠ j, the following holds: where \(p(\vec {x}\ \left |\ G\right )\) is the probability that a “good” applicant has attributes \(\vec {x}\). The probability of an applicant being “good” if she is characterized by the attributes \(\vec {x}\) can now be found by applying Bayes’ theorem: The probability of an applicant being “bad” if she is characterized by the attributes \(\vec {x}\) is

The attributes \(\vec {x}\) are typically converted into a score, \(s(\vec {x})\), which is such that \(p\left (G\ \right |\ \vec {x})=\ p\left (G\ \right |\ s(\vec {x}))\). A popular score function is the log-odds score [42]: where s _pop is the log of the relative proportion of “good” and “bad” applicants in the population and \(woe\left (\vec {x}\right )\) is the weight of evidence of the attribute combination \(\vec {x}\). Because of the conditional independence of the attributes, we can rewrite Eq. (17) as If \(woe\left (x_i\right )\) is equal to 0, then this attribute does not affect the estimation of the status of an applicant. The prior probabilities π _G and π _B are estimated using the proportions of good and bad applicants in the training sample; the same applies to the weight of evidence of the attributes, as illustrated in the example below.

A lender can therefore define a cutoff score, below which applicants are automatically rejected as “bad.” Usually, the score \(s(\vec {x})\) is linearly transformed so that its interpretation is more straightforward. The NB classifier performs relatively well in many applications but, according to Thomas et al. [42], it shows poor performance in the field of credit scoring. However, its most significant advantage is that it is easy to interpret, which is a property of growing importance in the industry.

Decision Trees (also known as Classification Trees) are a classification method that uses the training dataset to construct decision rules organized into tree-like structures, where each branch represents an association between the input values and the output label. Although different algorithms exist (such as classification and regression trees, also known as CART), we focus on the popular C4.5 algorithm developed by Quinlan [37]. At each node, the C4.5 algorithm splits the training dataset according to the most influential feature through an iterative process. The most influential feature is the one with the lowest entropy (or, similarly, with the highest information gain). Let \(\hat {\pi }_G\) be the proportion of “good” applicants and \(\hat {\pi }_B\) the proportion of “bad” applicants in the sample S. The entropy of S is then defined as in Baesens et al. [5]: According to this formula, the maximum value of the entropy is equal to 1 when \(\hat {\pi }_G = \hat {\pi }_B = 0.5\) and it is minimal at 0, which happens when either \(\hat {\pi }_G=0\) or \(\hat {\pi }_B=0\). In other words, an entropy of 0 means that we have been able to identify the characteristics that lead to a group of good (bad) applicants. In order to split the sample, we compute the gain ratio: \(Gain\left (S, x_i\right )\) is the expected reduction in entropy due to splitting the sample according to feature x _i and it is calculated as where υ ∈values(x _i), S _υ is a subset of the individuals in S that share the same value of the feature x _i, and where k ∈values(x _i) and S _k is a subset of the individuals in S that share the same value of the feature x _i. The latter term represents the entropy of S relative to the feature x _i. Once such a tree has been constructed, we can predict the probability that a new applicant will be a “bad” one using the proportion of “bad” customers in the leaf that corresponds to the applicant’s characteristics.

NN models were initially inspired by studies of the human brain [8, 9]. A NN model consists of input, hidden, and output layers of interconnected neurons. Neurons in one layer are combined through a set of weights and fed to the next layer. In its simplest single-layer form, a NN consists of an input layer (containing the applicants’ characteristics) and an output layer. More precisely, a single-layer NN is modeled as follows: where x ₁, …, x _n are the applicant’s characteristics, which in a NN are typically referred to as signals, ω _k1, …, ω _kn are the weights connecting characteristic i to the layer k (also called synaptic weights), and ω _k0 is the “bias” (which plays a similar role to the intercept term in a linear regression). Eq. (23) describes a single-layer NN, so that k = 1. A positive weight is called excitatory because it increases the effect of the corresponding characteristic, while a negative weight is called inhibitory because it decreases the effect of a positive characteristic [42]. The function f that transform the inputs into the output is called activation function and may take a number of specifications. However, in binary classification problems, it may be convenient to use a logistic function, as it produces an output value in the range [0, 1]. A cut-off value is applied to y _k to decide whether the applicant should be classified as good or bad. Figure 2 illustrates how a single-layer NN works.

A single-layer NN model shows a satisfactory performance only if the classes can be linearly separated. However, if the classes are not linearly separable, a multilayer model could be used [33]. Therefore, in the rest of this section, we describe multilayer perceptron (MLP) models, which are the most popular NN models in classification problems [5]. According to Bishop [9], even though multiple hidden layers may be used, a considerable number of papers have shown that MLP NN models with one hidden layer are universal nonlinear discriminant functions that can approximate arbitrarily well any continuous function. An MLP model with one hidden layer, which is also called a three-layer NN, is shown in Fig. 3. This model can be represented algebraically as where f ⁽¹⁾ is the activation function on the second (hidden) layer and y _k for k = 1 …, r are the outputs from the hidden layer that simultaneously represent the inputs to the third layer. Therefore, the final output values z _v can be written as where f ⁽²⁾ is the activation function of the third (output) layer, z _v for v = 1, …, s are the final outputs, and K _vk are the weights applied to the y _k values. The estimation of the weights is called training of the model and to this purpose the most popular method is the back-propagation algorithm, in which the pairs of input values and output values are presented to the model many times with the goal of finding the weights that minimize an error function [42].

The SVM method was initially developed by Vapnik [43]. The idea of this method is to transform the input space into a high-dimensional feature space by using a nonlinear function φ(•). Then, a linear classifier can be used to distinguish between “good” and “bad” applicants. Given a training dataset of N pairs of observations \({\left (\vec {x}_i, y_i\right )}_{i=1}^N\), where \(\vec {x}_i\) are the attributes of customer i and y _i is the corresponding binary label, such that y _i ∈ [−1,  + 1], the SVM model should satisfy the following conditions: which is equivalent to The above inequalities construct a hyperplane in the feature space, defined by \(\left \{x|\vec {w}^T\varphi \left (\vec {x}_i\right )+b=0\right \}\), which distinguishes between two classes (see Fig. 4 for the illustration of a simple two-dimensional case). The observations on the lines \(\vec {w}^T\varphi \left (x_i\right )+b=1\) and \(\vec {w}^T\varphi \left (x_i\right )+b=-1\) are called the support vectors. The parameters of the separating hyperplane are estimated by maximizing the perpendicular distance (called the margin), between the closest support vector and the separating hyperplane while at the same time minimizing the misclassification error.

The optimization problem is defined as: where the variables ξ _i are slack variables and C is a positive tuning parameter [5]. The Lagrangian to this optimization problem is defined as follows: The classifier is obtained by minimizing \(\mathscr {L}\left (\vec {w}, b, \xi ; \vec {\alpha }, \vec {\nu }\right )\) with respect to \(\vec {w}, b,\xi \) and maximizing it with respect to \(\vec {\alpha }\), \(\vec {\nu }\). In the first step, by taking the derivatives with respect to \(\vec {w},b,\xi \), setting them to zero, and exploiting the results, one may represent the classifier as where \(K\left (\vec {x}_i,\vec {x}\right )={\varphi \left (\vec {x}_i\right )}^T\varphi \left (\vec {x}_i\right )\) is computed using a positive-definite kernel function. Some possible kernel functions are the radial basis function \(K\left (\vec {x}_i, \vec {x}\right )=exp({-||x_i-x_j||{ }_2^2}/{\sigma ^2})\) and the linear function \(K\left (\vec {x}_i, \vec {x}\right )=\vec {x}_i^T\vec {x}_j.\) At this point, the Lagrange multipliers α _i can be found by solving:

In the k-NN method, any new applicant is classified based on a comparison with the training sample using a distance metric. The approach consists of calculating the distances between the new instance that needs to be classified and each instance in the training sample that has been already classified and selecting the set of the k-nearest observations. Then, the class label is assigned according to the most common class among k-nearest neighbors using a majority voting scheme or a distance-weighted voting scheme [41]. One major drawback of the k-NN method is that it is extremely sensitive to the choice of the parameter k, as illustrated in Fig. 5. Given the same dataset, if k=1 the new instance is classified as “bad,” while if k=3 the neighborhood contains one “bad” and two “good” applicants, thus, the new instance will be classified as “good.” In general, using a small k leads to overfitting (i.e., excessive adaptation to the training dataset), while using a large k reduces accuracy by including data points that are too far from the new case [41].

The most common choice of a distance metric is the Euclidean distance, which can be computed as: where \(\vec {x}_i\) and \(\vec {x}_j\) are the vectors of the input data of instances i and j, respectively. Once the distances between the newest and every instance in the training sample are calculated, the new instance can be classified based on the information available from its k-nearest neighbors. As seen above, the most common approach is to use the majority class of k-nearest examples, the so-called majority voting approach where y ^new is the class of the new instance, ν is a class label, \(\vec {S}_k\) is the set containing k-closest training instances, y _i is the class label of one of the k-nearest observations, and I(•) is a standard indicator function.

The major drawback of the majority voting approach is that it gives the same weight to every k-nearest neighbor. This makes the method very sensitive to the choice of k, as discussed previously. However, this problem might be overcome by attaching to each neighbor a weight based on its distance from the new instance, i.e., This approach is known as the distance-weighted voting scheme, and the class label of the new instance can be found in the following way: One of the main advantages of k-NN is its simplicity. Indeed, its logic is similar to the process of traditional credit decisions, which were made by comparing a new applicant with similar applicants [10]. However, because estimation needs to be performed afresh when one is to classify a new instance, the classification speed may be slow, especially with large training samples.

GA are heuristic, combinatorial optimization search techniques employed to determine automatically the adequate discriminant functions and the valid attributes [35]. The search for the optimal solution to a problem with GA imitates the evolutionary process of biological organisms, as in Darwin’s natural selection theory. In order to understand how a GA works in the context of credit scoring, let us suppose that (x ₁, …, x _n) is a set of attributes used to decide whether an applicant is good or bad according to a simple linear function: Each solution is represented by the vector \(\vec {\beta }=(\beta _0,\beta _1, \ldots , \beta _N)\) whose elements are the coefficients assigned to each attribute. The initial step of the process is the generation of a random population of solutions \(\vec {\beta }_{J}^{0}\) and the evaluation of their fitness using a fitness function. Then, the following algorithms are applied: The application of these algorithms results in the generation of a new population of solutions \(\vec {\beta }_{J}^{0}\). The algorithms selection-crossover-mutation are applied recursively until an (approximate) optimal solution \(\vec {\beta }_{J}^\ast \) is converged to.

Compared to traditional statistical approaches and NN, GA offers the advantage of not being limited in its effectiveness by the form of functions and parameter estimation [11]. Furthermore, GA is a nonparametric tool that can perform well even in small datasets [34].

In order to improve the accuracy of the individual (or base) classifiers illustrated above, ensemble (or classifier combination) methods are often used [41]. Ensemble methods are based on the idea of training multiple models to solve the same problem and then combine them to get better results. The main hypothesis is that when weak models are correctly combined, we can obtain more accurate and/or robust models. In order to understand why ensemble classifiers may reduce the error rate of individual models, it may be useful to consider the following example.

It is easy to understand that ensemble classifiers perform especially well when they are uncorrelated. Although in real-world applications it is difficult to obtain base classifiers that are totally uncorrelated, considerable improvements in the performance of ensemble classifiers are observed even when some correlations exists but are low [17]. Ensemble models can be split into homogeneous and heterogeneous. Homogeneous ensemble models use only one type of classifier and rely on resampling techniques to generate k different classifiers that are then aggregated according to some rule (e.g., majority voting). Examples of homogeneous ensemble models are bagging and boosting methods. More precisely, the bagging algorithm creates k bootstrapped samples of the same size as the original one by drawing with replacement from the dataset. All the samples are created in parallel and the estimated classifiers are aggregated according to majority voting. Boosting algorithms work in the same spirit as bagging but the models are not fitted in parallel: a sequential approach is used and at each step of the algorithm the model is fitted, giving more importance to the observations in the training dataset that were badly handled in the previous iteration. Although different boosting algorithms are possible, one of the most popular is AdaBoost. AdaBoost was first introduced by Freund and Schapire [19]. This algorithm starts by calculating the error of a base classifier h _t: Then, the importance of the base classifier h _t is calculated as: The parameter α _t is used to update the weights assigned to the training instances. Let \(\omega _i^{(t)}\) be the weight assigned to the training instance i in the t ^th boosting round. Then, the updated weight is calculated as: where Z _t is the normalization factor, such that \(\sum _{i}{\omega _i^{(t+1)}=1}\). Finally, the AdaBoost algorithm decision is based on In contrast to homogeneous ensemble methods, heterogeneous ensemble methods combine different types of classifiers. The main idea behind these methods is that different algorithms might have different views on the data and thus combining them helps to achieve remarkable improvements in predictive performance [47]. An example of heterogeneous ensemble method can be the following: A comparative evaluation of alternative ensemble methods is provided in Sect. 4.2.

In the first decade of the 2000s, the focus of most papers had been on performing comparisons among individual classifiers. Understandably, the question of whether advanced methods of classification, such as NN and SVM, might outperform LR and LDA had attracted much attention. While some authors have since then concluded that NN classifiers are superior to both LR and LDA (see, e.g., [2]), generally, it has been shown that simple linear classifiers lead to a satisfactory performance and, in most cases, that the differences between NN and LR are not statistically significant [5]. This section compares the findings of twelve papers concerning individual classifiers in the field of credit scoring. Papers were selected based on two features: first, the number of citations, and, second, the publishing date. The sample combines well-known papers (i.e., [45, 5]) with recent work (e.g., [29, 3]) in an attempt to provide a well-rounded overview.

One of the first comprehensive comparisons of linear methods with more advanced classifiers was West [45]. He tested five NN models, two parametric models (LR, LDA), and three nonparametric models (k-NN, kernel density, and DT) on two real-world datasets. He found that in the case of both datasets, LR led to the lowest credit scoring error, followed by the NN models. He also found that the differences in performance scores of the superior models (LR and three different way to implement NN) vs. the outperformed models were not statistically significant. Overall, he concluded that LR was the best choice among individual classifiers he tested. However, his methodology presented a few drawbacks that made some of his findings potentially questionable. First, West [45] used only one method of performance evaluation and ranking, namely, average scoring accuracy. Furthermore, the size of his datasets was small, containing approximately 1700 observations in total (1000 German credit applicants, 700 of which were creditworthy, and 690 Australian applicants, 307 of which were creditworthy).

Baesens et al. [5] remains one of the most comprehensive comparisons of different individual classification methods. This paper overcame the limitations in West [45] by using eight extensive datasets (for a total of 4875 observations) and multiple evaluation methods, such as the percentage of correctly classified cases, sensitivity, specificity, and the area under the receiver operating curve (henceforth, AUC, an accuracy metric that is widely used when evaluating different classifiers).³ However, the results reported by Baesens et al. [5] were similar to West’s [45]: NN and SVM classifiers had the best average results; however, also LR and LDA showed a very good performance, suggesting that most of the credit datasets are only weakly nonlinear. These results have found further support in the work of Lessmann et al. [29], who updated the findings in [5] and showed that NN models perform better than LR model, but only slightly.⁴

These early papers did not contain any evidence on the performance of GA. One of the earliest papers comparing genetic algorithms with other credit scoring models is Yobas et al. [49], who compared the predictive performance of LDA with three computational intelligence techniques (a NN, a decision tree, and a genetic algorithm) using a small sample (1001 individuals) of credit scoring data. They found that LDA was superior to genetic algorithms and NN. Fritz and Hosemann [20] also reached a similar conclusion even though doubts existed on their use of the same training and test sets for different techniques. Recently, these early results have been overthrown. Ong et al. [35] compared the performance of genetic algorithms to MLP, decision trees (CART and C4.5), and LR using two real-world datasets, which included 1690 observations. Genetic algorithms turned out to outperform other methods, showing a solid performance even on relatively small datasets. Huang et al. [26] compared the performance of GA against NN, SVM, and decision tree models in a credit scoring application using the Australian and German benchmark data (for a total of almost 1700 credit applicants). Their study revealed superior classification accuracy from GA than under other techniques, although differences are marginal. Abdou [1] has investigated the relative performance of GA using data from Egyptian public sector banks, comparing this technique with probit analysis, reporting that GA achieved the highest accuracy rate and also the lowest type-I and type-II errors when compared with other techniques.

One more recent and comprehensive study is that of Finlay [16], who evaluated the performance of five alternative classifiers, namely, LR, LDA, CART, NN, and k-NN, using the rather large dataset of Experian UK on credit applications (including a total of 88,789 applications, 13,261 of which were classified as “bad”). He found that the individual model with the best performance is NN; however, he also showed that the overperformance of nonlinear models over their linear counterparts is rather limited (in line with [5]).

Starting in 2010, most papers have shifted their focus to comparisons of the performance of ensemble classifiers, which are covered in the next section. However, some recent studies exist that evaluate the performance of individual classifiers. For instance, Ala’raj and Abbod [2] (who used five real-world datasets for a total of 3620 credit applications) and Bequé and Lessmann [7] (who used three real-world credit datasets for a total of 2915 applications) have found that LR has the best performance among the range of individual classifiers they considered. Although ML approaches are better at capturing nonlinear relationships, similarly to what is typical in credit risk applications (see [4]), it could be concluded that, in general, a simple LR model remains a solid choice among individual classifiers.

According to Lessmann et al. [29], the new methods that have appeared in ML have led to superior performance when compared to individual classifiers. However, only a few papers concerning credit scoring have examined the potential of ensemble methods, and most papers have focused on simple approaches. This section attempts to determine whether ensemble classifiers offer significant improvements in performance when compared to the best available individual classifiers and examines the issue of uncovering which ensemble methods may provide the most promising results. To succeed in this objective, we have selected and surveyed ten key papers concerning ensemble classifiers in the field of credit scoring.

West et al. [46] were among the first researchers to test the relative performance of ensemble methods in credit scoring. They selected three ensemble strategies, namely, cross-validation, bagging, and boosting, and compared them to the MLP NN as a base classifier on two datasets.⁵ West and coauthors concluded that among the three chosen ensemble classifiers, boosting was the most unstable and had a mean error higher than their baseline model. The remaining two ensemble methods showed statistically significant improvements in performance compared to MLP NN; however, they were not able to single out which ensemble strategy performed the best since they obtained contrasting results on the two test datasets. One of the main limitations of this seminal study is that only one metric of performance evaluation was employed. Another extensive paper on the comparative performance of ensemble classifiers is Zhou et al.’s [51]. They compared six ensemble methods based on LS-SVM to 19 individual classifiers, with applications to two different real-world datasets (for a total of 1113 observations). The results were evaluated using three different performance measures, i.e., sensitivity, the percentage of correctly classified cases, and AUC. They reported that the ensemble methods assessed in their paper could not lead to results that would be statistically superior to an LR individual classifier. Even though the differences in performance were not large, the ensemble models based on the LS-SVM provided promising solutions to the classification problem that was not worse than linear methods. Similarly, Louzada et al. [30] have recently used three famous and publicly available datasets (the Australian, the German, and the Japanese credit data) to perform simulations under both balanced (p = 0:5, 50% of bad payers) and imbalanced cases (p = 0:1, 10% of bad payers). They report that two methods, SVM and fuzzy complex systems offer a superior and statistically significant predictive performance. However, they also notice that in most cases there is a shift in predictive performance when the method is applied to imbalanced data. Huang and Wu [25] report that the use of boosted GA methods improves the performance of underlying classifiers and appears to be more robust than single prediction methods. Marqués et al. [31] have evaluated the performance of seven individual classifier techniques when used as members of five different ensemble methods (among them, bagging and AdaBoost) on six real-world credit datasets using a fivefold cross-validation method (each original dataset was randomly divided into five stratified parts of equal size; for each fold, four blocks were pooled as the training data, and the remaining part was employed as the hold out sample). Their statistical tests show that decision trees constitute the best solution for most ensemble methods, closely followed by the MLP NN and LR, whereas the k-NN and the NB classifiers appear to be significantly the worst.

All the papers discussed so far did not offer a comprehensive comparison of different ensemble methods, but rather they focused on a few techniques and compared them on a small number of datasets. Furthermore, they did not always adopt appropriate statistical tests of equal classification performance. The first comprehensive study that has attempted to overcome these issues is Lessmann et al. [29], who have compared 16 individual classifiers with 25 ensemble algorithms over 8 datasets. The selected classifiers include both homogeneous (including bagging and boosting) and heterogeneous ensembles. The models were evaluated using six different performance metrics. Their results show that the best individual classifiers, namely, NN and LR, had average ranks of 14 and 16 respectively, being systematically dominated by ensemble methods. Based on the modest performance of individual classifiers, Lessmann et al. [29] conclude that ML techniques have progressed notably since the first decade of the 2000s. Furthermore, they report that heterogeneous ensemble classifiers provide the best predictive performance.

Lessmann et al. [29] have also examined the potential financial implications of using ensemble scoring methods. They considered 25 different cost ratios based on the assumption that accepting a “bad” application always costs more than denying a “good” application [42]. After testing three models (NN, RF, and HCES-Bag) against LR, Lessmann et al. [29] conclude that for all cost ratios, the more advanced classifiers led to significant cost savings. However, the most accurate ensemble classifier, HCES-Bag, on average achieved lower cost savings than the radial basis function NN method, 4.8 percent and 5.7 percent, respectively. Based on these results, they suggested that the most statistically accurate classifier may not always be the best choice for improving the profitability of the credit lending business.

Two additional studies, Florez-Lopez and Ramon-Jeronimo [18] and Xia et al. [48], have focused on the interpretability of ensemble methods, constructing ensemble models that can be used to support managerial decisions. Their empirical results confirmed the findings of Lessmann et al. [29] that ensemble methods consistently lead to better performances than individual scoring. Furthermore, they concluded that it is possible to build an ensemble model that has both high interpretability and a high accuracy rate. Overall, based on the papers considered in this section, it is evident that ensemble models offer higher accuracy compared to the best individual models. However, it is impossible to select one ensemble approach that will have the best performance over all datasets and error costs. We expect that scores of future papers will appear with new, more advanced methods and that the search for “the silver bullet” in the field of credit scoring will not end soon.

Another promising development in credit scoring concerns one-class classification methods (OCC), i.e., ML methods that try to learn from one class only. One of the biggest practical obstacles to applying scoring methods is the class imbalance feature of most (all) datasets, the so-called low-default portfolio problem. Because financial institutions only store historical data concerning the accepted applicants, the characteristics of “bad” applicants present in their data bases may not be statistically reliable to provide a basis for future predictions ([27]. Empirical and theoretical work has demonstrated that the accuracy rate may be strongly biased with respect to imbalance in class distribution and that it may ignore a range of misclassification costs [14], as in applied work it is generally believed that the costs associated with type-II errors (bad customers misclassified as good) are much higher than the misclassification costs associated with type-I errors (good customers mispredicted as bad). OCC attempts to differentiate a set of target instances from all the others. The distinguishing feature of OCC is that it requires labeled instances in the training sample for the target class only, which in the case of credit scoring are “good” applicants (as the number of “good” applicants is larger than that of “bad” applicants). This section surveys whether OCC methods can offer a comparable performance to the best two-class classifiers in the presence of imbalanced data features.

The literature on this topic is still limited. One of the most comprehensive studies is a paper by Kennedy [27], in which he compared eight OCC methods, in which models are separately trained over different classes of datasets, with eight two-class individual classifiers (e.g., k-NN, NB, LR) over three datasets. Two important conclusions emerged. First, the performance of two-class classifiers deteriorates significantly with an increasing class imbalance. However, the performance of some classifiers, namely, LR and NB, remains relatively robust even for imbalanced datasets, while the performance of NN, SVM, and k-NN deteriorates rapidly. Second, one-class classifiers show superior performance compared to two-class classifiers only at high levels of imbalance (starting at 99% of “good” and 1% of “bad” applicants). However, the differences in performance between OCC models and LR model were not statistically significant in most cases. Kennedy [27] concluded that OCC methods failed to show statistically significant improvements in performance compared to the best two-class classification methods. Using a proprietary dataset from a major US commercial bank from January 2005 to April 2009, Khandani et al. [28] showed that conditioning on certain changes in a consumer’s bank account activity can lead to considerably more accurate forecasts of credit card delinquencies by analyzing subtle nonlinear patterns in consumer expenditures, savings, and debt payments using CART and SVM compared to simple regression and logit approaches. Importantly, their trees are “boosted” to deal with the imbalanced features of the data: instead of equally weighting all the observations in the training set, they weight the scarcer observations more heavily than the more populous ones.

These findings are in line with studies in other fields. Overall, the conclusion that can be drawn is that OCC methods should not be used for classification problems in credit scoring. Two-class individual classifiers show superior or comparable performance for all cases, except for cases of extreme imbalances.