Vine Copula-Based Classifiers with Applications

It is important that good choices of univariate distributions are made for individual variables in each class and checked for adequacy of fit with Q-Q plots (for continuous variables). In previous applications, we have sometimes found the need for 3-parameter and 4-parameter families for variables with unimodal histograms and boxplots that show skewness and a variety of tail weights. These include skew-t (Jones & Faddy, 2003), skew-normal (Azzalini, 1985), and generalized gamma outlined in Section A.3 of Klugman et al. (2010). When no low-dimensional parametric family provides a good fit, kernel density estimation, e.g., R package kde1d (Nagler & Vatter, 2022a), can be used after variables have been transformed to avoid extreme skewness.

Assume the training data set consists of \(n_j\) observations for class j. Let \(\varvec{x}_{p,j} = (x_{1,p,j}, \ldots , x_{n_j,p,j})^\top \) denote the pth feature vector in the jth class for \(p=1,\ldots , d\), and \(j=1,\ldots , k\). For the vine copula-based classifier, after the set of joint density function \(\{{\widehat{f}}_j\}\) for d features are separately estimated by class for a training set, posterior probabilities for a feature vector \(\varvec{x}^*\) are based on relative densities using prior class probabilities \(\pi _j\):For a single class prediction, the vector \(\varvec{x}^*\) would be assigned to the class asIn most applications, it is best to use \(\pi _j=k^{-1}\) to avoid majority classes in the data have higher prediction probabilities (such as with non-discriminant classification methods).

The theoretical counterpart of Eq. 7 with \({\widehat{f}}_a\) replaced by \(f_a\) for \(a\in \{1,\ldots ,k\}\) is invariant to a common (over classes) monotone increasing transform of any variable because the Jacobian term for the transform cancels from the numerator and denominator. However, since Eq. 7 is not invariant to monotone transforms, good preprocessing transforms are needed with data to get more reliable density estimates.

Vine copula classifiers have been used in limited settings in some published articles. Chen (2014) restricts to the D-vine structure, which is a boundary class of regular vines and has a limited choice of parametric bivariate copula families; there is no detailed discussion of fitting univariate distributions. Carrera et al. (2016) also use the D-vine structure for a 5-class problem with a limited choice of parametric bivariate copula families and assume univariate Gaussian distributions for individual variables. Nagler and Czado (2016) have a 2-class example to illustrate the use of bivariate copulas based on kernel density estimation when scatterplots suggest that the usual parametric bivariate copula families are inadequate. Also, Schellhase and Spanhel (2018) showed the usage of non-simplified vine copulas in a 2-class classification. Carrera et al. (2019) also allows for all regular vines, but there are no details of exploratory data analysis and feature extraction based on images for their 2-class example of dunes and non-dunes.

The vine-based classification procedure can also be applied to a subset of d available features, or one can start with vine copula constructions based on \(3\le d_1<d\) features and add extra features if performance measures based on \(\{{\widehat{\pi }}_j(\varvec{x}^*)\}\), over \(\varvec{x}^*\) in a validation set, improve with additional features.

In addition to the above discussion of the estimation of vine copula models and univariate densities, there are issues with feature selection approaches and performance evaluations, especially for \(k\ge 3\). These are discussed in the following subsections.

Some features might not be needed for classifiers, i.e., excluding them improves the performance and leads to more parsimony and greater computational efficiency. We develop two feature selection approaches for vine copula-based classifiers — we refer to these as a filtering method and a wrapper method, using the terminology of Guyon and Elisseeff (2003). The former uses the (conditional) mutual information of features given classes and does not depend on the classification method. The latter uses the classifier’s accuracy at each iteration and performs sequential forward feature selection.

Filtering methods use the information in features without fitting any classification models. However, their performance varies by models and data sets (Bommert et al., 2020). We assess how well a filtering method does for vine copula-based classifiers for comparisons.

Let X be a discrete or continuous feature with probability mass function or density \(f_X\) and let Y be a discrete class label random variable with probability mass function \(f_Y\). Let their joint density be \(f_{XY}\). The mutual information for X and Y is given bywhere \(\nu \) is a product measure involving counting measure if X is discrete and Lebesgue measure if X is continuous. Equation 9 can be regarded as the amount of information the variable X has about Y. For applications, Eq. 9 is simpler to compute as a double summation of the continuous variable X discretized into an ordinal variable; otherwise, (kernel) density estimation is needed for \(f_X\) and \(f_{X|Y}\). Our experience is that the discretization tends to lead to smaller values but does not change the importance ranking of features. When X and Y are independent, e.g., the feature X is uninformative for classification, Eq. 9 is zero.

Next, let \(\varvec{Z}\) be a random vector of discrete and continuous random variables with density \(f_{\varvec{Z}}\) and suppose \((X, Y, \varvec{Z})\) has the joint density \(f_{XYZ}\) and marginal densities \(f_{XZ}\) and \(f_{XY}\). The conditional mutual information between X and Y given \(\varvec{Z}\) isIn comparison to the mutual information in Eq. 9, the conditional mutual information in Eq. 10 measures the information the variable X provides about the variable Y conditioned on \(\varvec{Z}\). If the set of features \(\varvec{Z}\) has been selected, Eq. 10 summarizes the information the additional feature X provides about the class label Y. Similar to Eq. 9, for applications, Eq. 10 is simpler to compute via a multiple sum if the features have been discretized into ordinal variables.

A filtering method might stop adding features to a classifier when it reaches the specified number of features. Alternatively, since conditional mutual information avoids choosing strongly associated features, another approach for stopping can be adding all features X whose conditional mutual information is higher than a specified threshold after \(\varvec{Z}\) reaches the dimension of 3.

We remark that different filtering methods, as analyzed in Bommert et al. (2020), can be applied and compared for vine copula-based classifiers.

Filtering features for classification might lead to different performances by models since filtering does not account for model fits. To overcome such a drawback and tailor the feature selection to a classifier, wrapper methods are applied; they consider how a classifier performs sequentially and iteratively. In this section, we propose Algorithm 1 as a wrapper for selecting features in vine copula-based classifiers.

We assume that each variable has been transformed to remove extreme skewness (and orders of magnitude variability) before exploratory data analysis and applications of classification methods. For the pth variable with values \(\varvec{x}_{p} = (\varvec{x}_{p,1}, \ldots , \varvec{x}_{p,k})^\top \) over k classes, we definewhere \({\hat{\mu }}_p, {\hat{\sigma }}_p\) are respectively the sample mean and standard deviation of \(\varvec{x}_{p}\) (values of all k classes). Next, a dissimilarity measure is defined and applied to these scaled values.

The algorithm depends on a dissimilarity measure m for two vectors of data values \(\varvec{a}\) and \(\varvec{b}\). We will use the simply defined measure based on the sample quartiles:where \({\widehat{F}}_a^{-1}\) and \({\widehat{F}}_b^{-1}\) are empirical quantile functions of \(\varvec{a}\) and \(\varvec{b}\), respectively. Thus, it measures the lack of overlap in the central 50% boxes of two side-by-side boxplots. If the central 50% box of one boxplot is nested within the central 50% box of the other, then this dissimilarity measure is small. Further, if there is a location shift from \(\varvec{a}\) to \(\varvec{b}\) so that quantiles shift by the same amount, \(m(\varvec{a},\varvec{b})\) is the same as the location shift. If the distribution of \(\varvec{a}\) is within the distribution of \(\varvec{b}\), and the two medians are the same, \(m(\varvec{a},\varvec{b})\) is close to 0.

The algorithm initially chooses two features for classification and then adds further features based on a criterion that uses information on misclassified observations based on previously selected features.

Even though a random choice for the initial selection is possible, there is no guarantee that the selected features are useful for classification. Instead, we propose to select the starting features by assessing the dissimilarity of values for the pth feature across k classes based on the evaluation of the sum \(M_p\) of \(m(\cdot ,\cdot )\) in Eq. 12 over all pairs of classes, i.e., \(M_p = \sum ^{k-1}_{i=1} \sum ^{k}_{j=i+1}m(\varvec{x}^s_{p,i}, \varvec{x}^s_{p,j})\) where \(\varvec{x}^s_{p,i} \) and \( \varvec{x}^s_{p,j} \) represent the values of the scaled \( p \)th feature in classes \( i \) and \( j \), respectively, as given in Eq. 11.

The first two selected features have the largest two values of \(M_p\). Over many classes, this choice matches the visualization of side-by-side boxplots of each feature over all classes. We may initially select more than two features, but starting with two ensures the classifier considers the pairwise dependence structure among features.

Additionally, the initial selection for our wrapper Algorithm 1 considers univariate dissimilarity across classes and might not reveal features that have marginal similarity across classes but still are important for classification, e.g., conditionally given another feature. For such cases, a filtering method can be considered.

However, Algorithm 1 is flexible in the dissimilarity measure. This measure could be defined in Eq. 12 or a modification, incorporating the sum of the absolute values of differences in the three quartiles. We recommend the following: first, examine some univariate and bivariate plots of variables by classes; then, develop a dissimilarity measure that can effectively quantify the class differences observed in these plots.

Vine copulas are then fitted to get \({\widehat{f}}_j\) for each class from which predicted class labels via Eq. 8 can be obtained, starting with the two selected features. Then, additional features are added one at a time until a stopping criterion holds.

Let the class label based on Eq. 8 be \(\gamma ^{(t)}_i\) for the ith observation in iteration t, which is compared with the true labels \(\{\gamma _i\}\) for all observations. We can identify the misclassified observations in class j and let \(\varvec{x}^{sMis, (t)}_{p,j}\) be the set of misclassified (scaled) values for the pth feature.

A pth feature with the large dissimilarity of \(\varvec{x}^{sMis, (t)}_{p,j}\) and \(\varvec{x}^{s}_{p,j'}\) for \(j\ne j'\), based on \(m(\cdot ,\cdot )\) in Eq. 12, is considered as an addition.

The \(p^{(t)}\)th feature which maximizes such sum of dissimilarity over relevant pairs of classes is selected for the next vine copula models with the selected feature set \(\mathcal {F}^{(t)}\). Hence, our wrapper algorithm aims to choose a feature through which our classifier may assign the misclassified observations of the previous iteration to their true class while preserving its current classification power.

Information or cross-validation-based criteria involving performance measures can be used to decide if a wrapper method-based classifier stops adding features.

Suppose there are k classes and the prediction probability vector of a classification method with d features for an out-of-sample case with feature vector \(\varvec{x}^*\) is \(\widehat{\varvec{\pi }}=\widehat{\varvec{\pi }}(\varvec{x}^*)=({\widehat{\pi }}_1,\ldots ,{\widehat{\pi }}_k)\) for the k classes based on Eq. 7. The point prediction of the class is based on the modal probability \(\gamma =\gamma (\varvec{x}^*)=\text {argmax}_j {\widehat{\pi }}_j\), as in Eq. 8. This summarization can lose information when comparing different methods because it does not take into account whether the modal probability is near 1 (easy discrimination) or near 1/k (difficult discrimination).

Let \((j_1,\ldots ,j_k)\) be a permutation of \((1,\ldots ,k)\) so that \({\widehat{\pi }}_{j_1}(\varvec{x}^*)\ge {\widehat{\pi }}_{j_2}(\varvec{x}^*) \ge \cdots \ge {\widehat{\pi }}_{j_k}(\varvec{x}^*)\). For \(k^{-1}<1-\alpha <1\) and level \(1-\alpha \), a \(100(1-\alpha )\)% categorical prediction interval with q classes derived from \(\widehat{\varvec{\pi }}(\varvec{x}^*)\) is the smallest ordered set \(J_q=(j_1,\ldots ,j_q)\) of classes for which \(\sum _{r=1}^q {\widehat{\pi }}_{j_r}(\varvec{x}^*)\ge 1-\alpha \).

Let \({\widehat{\pi }}_\gamma \) be the modal probability. Consider 50% and 80% prediction intervals:If the 50% interval has more than one class, it indicates that \(\varvec{x}^*\) is in the part of the feature space where more than one class have comparable density values. If the 80% interval has just one class, it indicates that \(\varvec{x}^*\) is in part of the feature space where one class has a much larger density than other classes.

In this section, we summarize some performance measures used to compare different classification methods or the same method with different subsets of features.

One summary of the classification performance for two classes is through a receiver operating characteristic (ROC) curve. The curve illustrates the true positive rate on the y-axis versus the false positive rate on the x-axis at different thresholds by a classification method. For instance, a random guess corresponds to a point on the diagonal line, whereas the perfect classification is represented by a horizontal line at \(y=1\). The area under the ROC curve, called AUC, is one numeric summary of the classification method’s performance. The multiclass AUC, called (m)AUC, is the average of the associated pairwise AUCs of the k classes (Hand & Till, 2001). The hardest pair to discriminate can be identified by the index of the minimum of such AUCs. Nevertheless, the AUC does not reflect the uncertainty in the classification in contrast to categorical prediction intervals defined in Sect. 3.3; see Sect. 4.3.1 for examples.

Hence, we introduce misclassification rates and average lengths of categorical prediction intervals. While the latter represents the uncertainty and difficulty of the discrimination, the former reflects the misclassification rates at different coverage levels. Let \(n_{j,test}\) be the number of cases in the test set with true class \(j_i\) and \(100(1-\alpha )\)% categorical prediction interval \(J_{1-\alpha ,i}\) for the ith case. Let \(n_{test}=\sum _{j=1}^k n_{j,test}\). For class j at level \(1-\alpha \), the misclassification rate is \([\sum _{i: j_i=j} I(j\not \in J_{1-\alpha ,i})]/n_{j,test}\), and the average length is \([\sum _{i: j_i=j} \texttt {cardinality}(J_{1-\alpha ,i})]/n_{j,test}\). Further, the maximum misclassification rate with the corresponding class highlights which class is hardest to discriminate at different levels.

Another performance measure is the out-of-sample negative log-likelihood score (Czado et al., 2009) given by \(-n_{test}^{-1}\sum _{i\in \text {test\ set}} \log {\widehat{\pi }}_{j_i}(\varvec{x}_i^*)\) when there are \(n_{test}\) cases in the test set, and the ith case has feature vector \(\varvec{x}_i^*\) and is in class \(j_i\).

Note that there is also some loss of information from numerical summaries compared with tables of categorical prediction intervals.

The number of observations ranges from 4459 in Classical to 5000 in Hip-Hop and Rap (see Table 7 in Appendix A). The estimated strength of the pairwise dependence measured by Kendall’s tau among continuous features varies by a pair of features and genre, as shown in Table 8 in Appendix A.

Figure 2 in Appendix A shows that the features acousticness, liveness, loudness, and speechiness are skewed for Rock, and a similar result applies to other classes. Since the univariate density estimation can be affected by skewness, we transform them using the logit transform for acousticness, liveness, and speechiness and a cube root transform for loudness after converting to all negative combined over all classes. We scale the popularity into [0,1] by dividing the values by 100. The resulting transformed features are referred to as tacoustic, tlive, tloud, tspeech, and tpopular (see Fig. 3 in Appendix A).

The binary variable, mode, can help discriminate Country songs from others as seen in Table 9 in Appendix A. Moreover, even though instrumentalness can take values between zero and one, it is zero for 30% of the songs, and some genres, such as Hip-Hop and Rap, have a smaller proportion of zeros. Therefore, we bin as [0, 0.00001], (0.00001, 0.20], (0.20, 1], calling the associated discrete variable with three levels as tinstrumental. Table 9 in Appendix A shows that Anime, Blues, Classical, Electronic, and Jazz are different from other genres regarding the distribution of tinstrumental.

Some pairs of classes might be hard to discriminate, making the classification problem hard. For its initial overview, we propose a procedure described in Appendix A. Accordingly, the overlap measure matrix in Table 10 in Appendix A shows that Rap and Hip-Hop are hard to discriminate, whereas Rap and Anime might be well separated. Additionally, Alternative has the highest overlap measure summed over other classes, showing that discriminating it from others is harder. The reverse applies to Classical.

In this section, we compare four discriminant analysis methods for \({\widehat{f}}_j\) in Eqs. 7 using equal class probabilities \(\pi _j\):

Fernández-Delgado et al. (2014) conclude that random forest-based approaches outperform hundreds of classifiers regarding classification accuracy, including neural networks, and Speiser et al. (2019) point out that the feature selection method proposed by Genuer et al. (2010) for random forests, called VSURF, usually performs better than other methods. Thus, we compare our vine copula-based classifier with random forests implemented in the R packages randomForest (Liaw & Wiener, 2002) and VSURF (Genuer et al., 2019). We work with the packages’ default specifications.

For ten genres and eight continuous features, a vine-based classifier and random forest (using all features) discriminate Classical better than the other genres as shown in Table 5, consistent with our explanatory data analysis. The methods’ misclassification rate for the 50% prediction interval for Classical and Rock is, on average, less than 17% and 24% percent, respectively. However, the associated average length for Classical is less than 1.26 for both methods, that for Rock is 2.19 for vines and 2.64 for random forest. Thus, the methods are more uncertain in the discrimination of Rock than that of Classical. In over half of the replicates, the random forest assigns the Rock’s observations into three genres in the 50% prediction intervals. We remark that there is a trade-off of shorter prediction intervals and slightly higher misclassification rates in the methods’ comparison.

Using eight continuous features, the average (m)AUC performances of random forests and vine copula-based classifiers are similar in the discrimination of four and ten genres in Table 6. Nevertheless, the vine method using the wrapper with a tuning parameter of zero is the best performer regarding the (m)AUC in both analyses. That method adds features in vine models until no further improvement in mAUC in a validation set is present. The same result applies while considering the minimum pairwise AUC. Further, all vine methods have a better minimum pairwise AUC than random forests. Thus, it shows that the vine methods discriminate the two classes {Hip-Hop, Rap} having similar density values better than random forests. Further, the feature of tlive was selected neither by our wrapper with zero tuning parameter nor by the random forest-specific variable selection, VSURF, in our replicates for the discrimination of ten genres. Thus, the tlive feature need not be recorded for song genre discrimination, and music experts should consider replacing it with another feature that can better discriminate between Hip-Hop and Rap.

VSURF has the best negative log-likelihood score in the discrimination of ten genres using eight continuous features. Thus, the VSURF discriminates the observations in their true classes with higher probabilities than the other methods. The same result holds for the vine method using the wrapper with a tuning parameter of zero among the vine-based classifiers for ten genre discrimination. However, the vine method with filtering four features is the best performer of the negative log-likelihood score in classifying four genres using eight continuous features. Moreover, VSURF performs better than filtering features with (conditional) mutual information for random forests in all cases regarding the three measures analyzed in Table 6.

The estimated dependence strength among pairs of features is mostly low in classes in Table 8 in Appendix A. However, as the strength increases, vine copula-based classifiers are more likely to outperform random forests similar to the prediction tasks, as shown in Sahin and Czado (2024), where the main interest is to predict a continuous response using vine copulas and selecting the relevant features. Thus, to analyze the impact of dependent features on random forests and the vine method, we run them using the three dependent features, tacoustic, energy, and tloud, in the discrimination of four genres. Then, the vine method, on average, discriminates classes better than random forests regarding the (m)AUC, minimum pairwise AUC, and negative log-likelihood scores as seen in Table 6. Even though three weakly dependent features, dance, tspeech, and valence, are considered in the discrimination, a vine-based classifier is better than a random forest. A reason might be that random forests need more features to discriminate the classes better.

While our wrapper with the zero tuning parameter selects 4.46 features, on average, in the discrimination of four classes, the number increases to 6.72 in the discrimination of ten classes out of eight features. Likewise, VSURF selects more features to discriminate ten classes than four classes. Hence, models in our data example need more features to discriminate more classes. Still, eliminating features improves the discrimination accuracy compared to using all features, as reflected in the negative log-likelihood score and minimum pairwise AUC by our wrapper algorithm with the zero tuning parameter.

On average, the computation time of vine and random forests is 4.19 and 1.90 s, respectively, using eight continuous features to classify ten genres. Moreover, in the same analysis, while the filtering on the vine takes less than one second, the wrapper runs around 9.52 s. Since the latter fits the vine models at each step, such computation time can be expected. For this case, the VSURF gives the highest average computation time of 38.20 s. Like our wrapper algorithm for vine-based classifiers, tailoring feature selection to a specific method comes with a trade-off of higher computational cost and better performances.

When considering two discrete variables, tinstrumental and mode, with eight continuous features, for the ten genres’ discrimination in vine-based classifiers, the multi AUC given in Table 6 increases by 0.01 through the selected features by our wrapper but not through filtering and using all features. The former is because (conditional) mutual information does not identify tinstrumental and mode within the most important four features in most replicates. While the average number of selected variables by our wrapper slightly increases with the inclusion of tinstrumental and mode, the wrapper method does not select mode at all and replaces tacoustic with tinstrumental in most replicates. Since tacoustic differs for Classical, Jazz, and Electronic but tinstrumental is different for five genres as shown in Appendix A, such a result can be expected. Further, our wrapper methods slightly improve the negative log-likelihood scores by adding tinstrumental. But, compared to using only eight continuous features, modeling ten features together decreases the vine method’s performance regarding the (m)AUC and negative log-likelihood score, on average. This shows the importance of selecting features for vine-based classifiers. Moreover, Appendix A shows that tinstrumental and mode have a similar distribution for Hip-Hop and Rap. Thus, adding tinstrumental and/or mode into vine-based classifiers does not impact their minimum pairwise AUC. Moreover, the misclassification rates still vary across genres, highlighting the distinct impact of tinstrumental on Blues, Jazz, and Electronic and that of mode on Country.

Discriminant analysis via the vine copula-based classifier is more interpretable than random forests. The vine copula-based classifier can lead to smooth functions of \(\{{\widehat{\pi }}_j(\varvec{x}): \varvec{x}\in \text{ feature } \text{ space }, j\in \{1,\ldots ,k\}\}\), and one could (maybe through projections) interpret different parts of the feature space that favors different classes. Random forests have good classification performance, but the conditional probability vector \(({\widehat{\pi }}_{RF}(j|\varvec{x}): j=1,\ldots ,k)\) is piecewise constant with a finite number of trees. Thus, if one attempts to study the behavior as a function of \(\varvec{x}\) by choosing out-of-sample vectors in a grid, one can see that the function can have many up-and-down jumps locally.