View selection in multi-view stacking: choosing the meta-learner

In the meta-learning problem, we have the binary outcome \(\varvec{y}\), and a matrix of cross-validated predictions \(\varvec{Z} = (z_{i}^{(v)}) \in [0,1]^{n \times V}\). Consider a multi-view stacking model where the final prediction is a simple linear combination of the base classifiers:We can obtain a so-called interpolating predictor (Breiman 1996) by computing the parameter estimates assubject to the constraints \(\beta _v \ge 0, v = 1, \dots , V\) and \(\sum _v \beta _v = 1\). The resulting prediction function interpolates in the sense that the final prediction \({\hat{y}}_i\) can never go outside the range of the predictions of the base classifiers \([\min _v {\hat{f}}_v(\varvec{x}^{(v)}_i), \max _v {\hat{f}}_v(\varvec{x}^{(v)}_i)]\) (Breiman 1996). Although originally proposed in the context of linear regression, it can also be used in binary classification, since if we have probabilistic base-classifiers making predictions in [0, 1], then the final prediction will also be in [0, 1]. Additionally the model is easy to interpret, since the final prediction is a weighted mean of the base classifiers’ predictions. Note that replacing the sum-to-one constraint with the constraint \(\sum _v \beta _v \le t\), with \(t \ge 0\) a tuning parameter, leads to the nonnegative lasso (Wu et al. 2014). The interpolating predictor can thus be thought of as a (linear) nonnegative lasso with a fixed amount of regularization.

Instead of a simple linear function to combine the base classifiers, we can also use the logistic function:with \(\beta _0 \in {\mathbb {R}}\) an intercept. The logistic function restricts predictions to [0, 1] without the need for constraints on the parameters. Parameter estimates in logistic regression are usually obtained through maximizing the log-likelihood or, equivalently, minimizing the negative log-likelihood. Denote by \(\varvec{\beta } = (\beta _1,..., \beta _V)^T\) the vector of regression coefficients, and by \(\varvec{z}_i = (z_{i}^{(1)}, z_{i}^{(2)}, \dots , z_{i}^{(V)})\) the ith row of \(\varvec{Z}\). Then the negative log-likelihood corresponding to the logistic regression model is given byAlthough constraints on the parameters are not required to keep the predictions in the range [0, 1], we can still employ regularization to perform view selection and obtain more stable models. The elastic net (Zou and Hastie 2005) is a popular regularization method which employs both \(L_1\) and \(L_2\) penalties. To obtain parameter estimates using the nonnegative variant of the elastic net, one optimizeswith tuning parameters \(\lambda \ge 0\) and \(\alpha \in [0,1]\). Ridge regression (\(\alpha = 0\)) and the lasso (\(\alpha = 1\)) are both special cases of the elastic net. Choosing any other value of \(\alpha\) leads to a mixture of \(L_1\) and \(L_2\) penalties. Since the columns of \(\varvec{Z}\) correspond to predictions of models trained using the same outcomes \(\varvec{y}\), it is very likely that at least some of them are highly correlated. When faced with a set of highly correlated views, the lasso may select one of them and discard the others. The addition of an \(L_2\) penalty causes the elastic net to favor solutions where the entire set of correlated views is included with moderate coefficients. Which type of behavior is desirable will depend on the research question at hand. In this paper we will apply ridge regression, the lasso, and the elastic net with \(\alpha = 0.5\), all with nonnegativity constraints. Note that, although ridge regression usually does not perform any view selection, the addition of nonnegativity constraints forces some coefficients to be zero, causing view selection even when \(\alpha = 0\).

The adaptive lasso (Zou 2006) is a weighted version of the lasso with data-dependent weights. Consider again the negative log-likelihood in (4). Let us define for each \(\beta _v \in \varvec{\beta }\) a corresponding weight \({\hat{w}}_v = 1 / |{\hat{\beta }}_v|^{\gamma }\), with \({\hat{\beta }}_v\) an initial estimate of \(\beta _v\), and \(\gamma > 0\) a tuning parameter. Then the adaptive lasso estimates are given byIn the context of linear models, Zou (2006) suggested to use OLS or ridge regression to obtain the initial \({\hat{\beta }}_v\)’s. We use (logistic) ridge regression with nonnegativity constraints to obtain the initial estimates. Due to the nonnegativity constraints, views which would have otherwise obtained a negative coefficient thus obtain an infinitely large penalty (i.e. are removed from the model) in the weighted lasso.

Stability selection is an ensemble learning framework originally proposed for use with the lasso (Meinshausen and Bühlmann 2010), although it can be used with a wide variety of feature selection methods (Hofner et al. 2015). The basic idea of stability selection is to apply a feature selection method on subsamples of the data, and then incorporate in the final model only those features which were chosen by the feature selection method on a sufficiently large proportion of the subsamples. In this study, we specifically use complementary pairs stability selection (Shah and Samworth 2013) with the nonnegative lasso, as implemented in the R package stabs (Hofner and Hothorn 2017). This procedure can be described as follows (Hofner et al. 2015; Hofner and Hothorn 2017; Shah and Samworth 2013): Typically B is set to 50, but the choice of q and \(\pi _{\text {thr}}\) is somewhat more involved. In particular, one can obtain a bound on the expected number of falsely selected variables, the so-called per-family error rate (PFER), for a given value of q and \(\pi _{\text {thr}}\) (Meinshausen and Bühlmann 2010; Shah and Samworth 2013; Hofner et al. 2015). For our particular choices of these parameters, see Sect. 4.2. The meta-classifier is obtained by fitting a nonnegative logistic regression model using only the cross-validated predictions corresponding to the set of stable views. Note that the computational cost of stability selection is several times larger than that of the regular lasso with cross-validation: in the linear case, stability selection has been estimated to be approximately 3 times more expensive than ten-fold cross-validation when \(n < V\), and approximately 5.5 times more expensive when \(n > V\) (Meinshausen and Bühlmann 2010).

Forward selection is a simple, greedy feature selection algorithm (Guyon and Elisseeff 2003). It is a so-called wrapper method, which means it can be used in combination with any learner (Guyon and Elisseeff 2003). The basic strategy is to start with a model with no features, and then add the single feature to the model which is “best” according to some criterion. One then proceeds to sequentially add the next “best” feature at every step until some stopping criterion is met. Here we consider forward selection based on the Akaike Information Criterion (AIC). In order to impose nonnegativity of the coefficients, we will use a slightly modified procedure which we will call nonnegative forward selection (NNFS). This procedure can be described as follows:

In order to compare the different meta-learners in terms of classification and view selection performance, we perform a series of simulations. We generate multi-view data with \(V = 30\) or \(V = 300\) disjoint views, where each view \(\varvec{X}^{(v)}, v = 1, \dots , V\), is an \(n \times m_v\) matrix of normally distributed features scaled to zero mean and unit variance. For each number of views, we consider two different view sizes. In any single simulated data set, all views are always set to be the same size. If \(V = 300\), then either \(m_v = 25\) or \(m_v = 250\). If \(V = 30\), then either \(m_v = 250\) or \(m_v = 2500\). We use two different sample sizes: \(n = 200\) or \(n = 2000\). In addition, we apply different correlation structures defined by the population correlation between features from the same view \(\rho _w\), and the population correlation between features from different views \(\rho _b\). We use six different parameterizations: (\(\rho _w = 0.1\), \(\rho _b = 0\)), (\(\rho _w = 0.5\), \(\rho _b = 0\)), (\(\rho _w = 0.9\), \(\rho _b = 0\)), (\(\rho _w = 0.5\), \(\rho _b = 0.4\)), (\(\rho _w = 0.9\), \(\rho _b = 0.4\)), and (\(\rho _w = 0.9\), \(\rho _b = 0.8\)). This leads to a total of \(2 \times 2 \times 2 \times 6 = 48\) different experimental conditions.

For each experimental condition, we simulate 100 multi-view data training sets. For each such data set, we randomly select 10 views. In 5 of those views, we determine all of the features to have a relationship with the outcome. In the other 5 views, we randomly determine 50% of the features to have a relationship with the outcome. The relationship between features and response is determined by a logistic regression model, where each feature related to the outcome is given a regression weight. In the setting with 30 views, we use the same regression weight as a similar simulation study in Van Loon et al. (2020). This regression weight is either 0.04 or \(-0.04\), each with probability 0.5. In the setting with 300 views, the number of features per view is reduced by a factor 10. To compensate for the reduction in the number of features, the aforementioned regression weights are multiplied by \(\sqrt{10}\) in this setting.

We apply multi-view stacking to each simulated training set, using logistic ridge regression as the base-learner. Once we obtain the matrix of cross-validated predictions \(\varvec{Z}\), we apply the seven different meta-learners. To assess classification performance, we generate a matching test set of 1000 observations for each training set, and calculate the classification accuracy of the stacked classifiers on this test set. To assess view selection performance we calculate three different measures: (1) the true positive rate (TPR), i.e. the average proportion of views truly related to the outcome that were correctly selected by the meta-learner; (2) the false positive rate (FPR), i.e. the average proportion of views not related to the outcome that were incorrectly selected by the meta-learner; and (3) the false discovery rate (FDR), i.e. the average proportion of the selected views that are not related to the outcome.

Although we can average over the 100 replications within each condition, with 7 different meta-learners and 48 experimental conditions, this would still lead to 336 averages for each of the outcome measures. In our reporting of the results we will therefore focus only on the most important interactions between the meta-learners and the different experimental conditions. To determine which interactions are most important in a data-driven way, we perform a mixed analysis of variance (ANOVA), and calculate a standardized measure of effect size (partial \(\eta ^2\)) for each interaction. We do this separately for each of the four outcome measures. A common rule of thumb is that \(\eta ^2 \ge 0.06\) corresponds to a moderate effect size, and \(\eta ^2 \ge 0.14\) corresponds to a large effect size (Cohen 1988; Rovai et al. 2013). We discuss only the interactions that have at least a moderate effect size \(\eta ^2 \ge 0.06\). Note that we use ANOVA only to calculate a measure of effect size, and do not report test statistics or p-values. This is because (1) these tests would rely too heavily on the assumptions of ANOVA, and (2) in a simulation study any arbitrarily small difference can be artificially made “significant" by increasing the number of replications.

All simulations are performed in R (version 3.4.0) (R Core Team 2017) on a high-performance computing cluster running Ubuntu (version 14.04.6 LTS) with Open Grid Scheduler/Grid Engine (version 2011.11p1). All pseudo-random number generation is performed using the Mersenne Twister (Matsumoto and Nishimura 1998), R’s default algorithm. The training of the base-learners, and the generation of cross-validated predictions is performed using an early development version of package mvs (Van Loon 2022). Optimization of the nonnegative ridge, elastic net, and lasso is performed using coordinate descent through the package glmnet 1.9–8 (Friedman et al. 2010). Nonnegativity constraints are implemented by setting a coefficient to zero if it becomes negative during the update cycle (Friedman et al. 2010; Hastie et al. 2015). To select the tuning parameter \(\lambda\), a sequence of 100 candidate values of \(\lambda\) is adaptively chosen by the software (Friedman et al. 2010). In particular, the 100 candidate values are decreasing on a log scale from \(\lambda _{\text {max}}\) to \(\lambda _{\text {min}}\), where \(\lambda _{\text {max}}\) is the smallest value such that the entire coefficient vector is zero, and \(\lambda _{\text {min}} = \epsilon \lambda _{\text {max}}\), with \(\epsilon = 10^{-4}\). The value of \(\lambda\) is then selected by minimizing the 10-fold cross-validation error. For the nonnegative elastic net we set \(\alpha = 0.5\). We choose to fix the value of \(\alpha\) at 0.5 rather than tune it so that we can compare the performance of the equal mixture of the two penalties with using only an \(L_1\) or only an \(L_2\) penalty. The code used for fitting the nonnegative adaptive lasso is also based on glmnet, were we use 10-fold cross-validation to select both \(\lambda\) and \(\gamma\). For \(\gamma\) we consider the possible values \(\{ 0.5, 1, 2\}\), for each of which we fit a path of 100 candidate values for \(\lambda\). We then choose the combination of \(\gamma\) and \(\lambda\) which has the lowest cross-validation error. For stability selection we use the package stabs 0.6–3 (Hofner and Hothorn 2017; Hofner et al. 2015). We adopt the recommendations of Hofner et al. (2015) for choosing the parameters, by specifying q and a desired bound \(\textit{PFER}_{\text {max}}\), and then calculating the associated threshold \(\pi _{\text {thr}}\). The parameter q should be chosen large enough that in theory all views corresponding to signal can be chosen (Hofner et al. 2015). We therefore choose \(q = 10\), since we have 10 views corresponding to signal. Note that this means that the procedure has additional information about the true model unavailable to the other meta-learners. We choose a desired bound of \(\textit{PFER}_{\text {max}} = 1.5\), which is equivalent to controlling the per-comparison error rate (PCER) at \(1.5 / 30 = 0.05\) when \(V = 30\), or \(1.5 / 300 = 0.005\) when \(V= 300\). Under the unimodality assumption of Shah and Samworth (2013), this leads to \(\pi _{\text {thr}} = 0.9\) when \(V = 30\), and \(\pi _{\text {thr}} = 0.57\) when \(V = 300\). The code used to perform nonnegative forward selection is based on stepAIC from MASS 7.3–47 (Venables and Ripley 2002). The optimization required for fitting the interpolating predictor is performed using the package lsei 1.2–0 (Wang et al. 2017). After optimization, coefficients smaller than \(\max (10^{-2}/V, 10^{-8})\) are set to zero.

We apply MVS with the seven different meta-learners to two gene expression data sets, namely the colitis data of Burczynski et al. (2006), and the breast cancer data of Ma et al. (2004). These data sets were previously used to compare the group lasso with the sparse group lasso (Simon et al. 2013), and to compare the group lasso with StaPLR (Van Loon et al. 2020). The colitis data (Burczynski et al. 2006) consists of 85 colitis cases and 42 healthy controls for which gene expression data was collected using 22,283 probe sets. As in (Van Loon et al. 2020), we matched this data to the C1 cytogenetic gene sets from MSigDB 6.1 (Subramanian et al. 2005), and removed any duplicate probes, genes not included in the C1 gene sets, and gene sets which consisted of only a single gene after matching. This led to a multi-view data set consisting of 356 views (gene sets), with an average view size of 33 features (genes). A boxplot of the distribution of the view sizes is included in Appendix 1. The total number of features was 11,761. All features were \(\text {log}_2\)-transformed, then standardized to zero mean and unit variance before applying the MVS procedure.

The breast cancer data (Ma et al. 2004) consists of 60 tumor samples labeled according to whether cancer did (28 cases) or did not (32 cases) recur. The data was matched to the C1 gene sets using the same procedure as in the colitis data, leading to a multi-view data set of 354 views, with an average view size of 36 features. A boxplot of the distribution of the view sizes is included in Appendix 1. The total number of features was 12,722. The features were already \(\text {log}_2\)-transformed, but were further standardized to zero mean and unit variance before applying the MVS procedure. To assess classification performance for each of the data sets, we perform 10-fold cross-validation. We repeat this procedure 10 times and average the results to account for random differences in the cross-validation partitions. The hold-out data in the cross-validation procedure is used only for model evaluation, not for parameter tuning. We again report classification accuracy using a standard threshold of 0.5. Because the colitis data is somewhat unbalanced in terms of class membership, one might additionally be interested in the performance of the methods across multiple possible thresholds. Due to different opinions regarding which metric is most suitable for comparing performance across multiple thresholds (Hand 2009; Flach et al. 2011; Hernández-Orallo et al. 2012), we report two popular metrics, namely the area under the receiver operating characteristic curve (AUC), and the H measure (Hand 2009). Both metrics can take values in \([0,1]\), but the AUC more typically takes values in \([0.5,1]\), with a value of 0.5 denoting a noninformative classifier. For the H measure, a noninformative classifier is associated with a value of zero.

In terms of view selection, each of the \(10 \times 10\) fitted models is associated with a set of selected views. However, quantities like TPR, FPR and FDR cannot be computed since the true status of the views is unknown. We therefore report the number of selected views, since this allows assessment of model sparsity. In addition, we report a measure of the stability of the set of selected views. In particular, we use the feature selection stability measure of Nogueira et al. (2018). This stability measure, \({\hat{\Phi }}\), has an upper bound of 1 and a lower bound which is asymptotically zero but depends on the number of fitted models (in our case it is approximately − 0.01). Higher values indicate increased stability, and \({\hat{\Phi }}\) attains its maximum of 1 if-and-only-if the set of selected views is the same for each fitted model (Nogueira et al. 2018). A particularly desirable property of this stability measure is that it is corrected for chance in the sense that its expected value is zero if a selection algorithm would select sets of views of a certain size randomly rather than systematically (Nogueira et al. 2018). The measure can additionally be considered a special case of the Fleiss’ Kappa (Fleiss 1971) measure of inter-rater agreement, where each of the fitted models is a “rater” classifying the views into whether or not they are relevant (Nogueira et al. 2018). Rules of thumb for interpreting the strength of agreement associated with a certain value of Fleiss’ Kappa have been formulated by Landis and Koch (1977):.00-.20=slight, 0.21\(-\)0.40=fair, 0.41\(-\)0.60=moderate, 0.61\(-\)0.80=substantial, 0.81\(-\)1.0=almost perfect. However, these rules of thumb are largely arbitrary (Landis and Koch 1977), and we will focus on relative comparisons.

We use the same software as described in Sect. 4.2. All cross-validation loops used for parameter tuning are nested within the outer loop used for evaluating classification performance. We again use the recommendations of Hofner et al. (2015) for choosing the parameters, by specifying q and a desired bound \(\textit{PFER}_{\text {max}}\), and then calculating the associated threshold \(\pi _{\text {thr}}\). We again specify a desired bound of 1.5. The parameter q should be large enough so that all views corresponding to signal can be selected (Hofner et al. 2015), but in real data the number of views corresponding to signal is unknown. However, domain knowledge or results from previous experiments can be used to obtain an estimate of the number of views corresponding to signal, and thus assist in choosing the value of q. The colitis and breast cancer data sets have previously been analyzed using StaPLR with the nonnegative lasso as a meta-learner (Van Loon et al. 2020). We set the value of q to the maximum number of views selected by the StaPLR method in the colitis and breast cancer data sets as reported in Van Loon et al. (2020). This leads to \(q = 15\), \(\pi _{\text {thr}} = 0.62\) for the colitis data, and \(q = 11\), \(\pi _{\text {thr}} = 0.57\) for the breast cancer data. Note that this means the stability selection procedure has access to information from a previous experiment that the other meta-learners do not have access to. The AUC is calculated using the package AUC 0.3.0 (Ballings and Van 2013), and the H measure using the package hmeasure 1.0–2 (Anagnostopoulos and Hand 2019). The stability measure \({\hat{\Phi }}\) was calculated using a custom script, which is included in the supplementary materials.

The results of applying MVS with the seven different meta-learners to the colitis data can be observed in Table 2. In terms of raw test accuracy the nonnegative lasso is the best performing meta-learner, followed by the nonnegative elastic net and the nonnegative adaptive lasso. In terms of AUC and H, the best performing meta-learners are the elastic net and ridge regression. However, the elastic net selects on average almost 4 times as many views as the lasso, and nonnegative ridge regression selects on average almost 13 times as many views, and both have lower raw test accuracy than the lasso. Since feature selection often comes at a cost in terms of stability (Xu et al. 2012), it is to be expected that view selection stability (\({\hat{\Phi }}\)) is higher for meta-learners that select more views. The results of two meta-learners do not align with this pattern, namely those for the interpolating predictor and NNFS. The interpolating predictor is very dense but has lower stability than several sparse models, while NNFS is less sparse than stability selection and also less stable. Note that we mean sparsity in terms of the number of selected views, but this corresponds to sparsity in terms of the number of selected features (see Appendix 1, Table 4).

The results for the breast cancer data can be observed in Table 3. The interpolating predictor and the lasso are the best performing meta-learners in terms of all three classification measures, with the interpolating predictor having higher test accuracy and H, and the lasso having higher AUC. However, the interpolating predictor selects over 80 times as many views as the lasso, and is less stable. Again, the interpolating predictor and NNFS do not align with the pattern that less sparsity is associated with higher stability.