Old but Gold or New and Shiny? Comparing Tree Ensembles for Ordinal Prediction with a Classic Parametric Approach

The cumulative model (McCullagh, 1980) is a parametric model which assumes that the observed ordinal response manifests an underlying latent variable that is continuous and can only be observed via thresholds that define the response categories. It models the probability \(P(Y \le r | \varvec{x})\) that the ordinal outcome variable Y for an observation with covariate vector \(\varvec{x}\) takes at most category \(r = 1, \dots , k\) aswhere \(-\infty< \gamma _1< \dots < \gamma _k = \infty \) denote the thresholds and F is a strictly increasing distribution function (Tutz, 2022). A common choice for F is the logistic function resulting in the proportional odds model (Tutz, 2022), i.e.,which will also be used in this work. For our simulation, we fitted the proportional odds model using the clm function from the ordinal package (Christensen, 2022). As such, we will be referring to proportional odds models in the context of ordinal prediction as CLM (cumulative link model) in the remainder of this work.

To compare the predictive performance of the different prediction methods, we have performed a simulation study in which we generated datasets of varying characteristics. All data were generated from proportional odds models as is a common choice for simulating ordinal data (e.g., Janitza et al., 2016; Hornung, 2019) and aligns with the ordinal prediction methods compared in this work, most of which implicitly or explicitly assume proportional odds settings. We varied the number of observations, n, between 250, 750, and 2500. As many ordinal data applications originate from (bio-)medicine, psychology or the social sciences, we aimed to keep a mix of medium sample sizes to create realistic scenarios (see, e.g., Shen et al., 2011, for a review of sample sizes in psychology). The number of covariates, p, was either 10 or 35. While the former setting reflects common application scenarios (cf. the real data examples studied further below), the latter setting represents a compromise between including a higher number of covariates and not putting the CLM which is known to suffer from high dimensionality (Zahid & Tutz, 2013) too much at a disadvantage. We would expect values such as 35 to be more commonly encountered when analyzing, e.g., large-scale assessment studies (see, e.g., Immekus et al., 2022). As increasing the number of covariates to 35 only introduced further noise variables, the two settings also distinguished between a setting with a high signal-to-noise-ratio (SNR) and a setting with a low SNR. The number of categories k was varied between 3, 5, and 7, as these commonly occur, e.g., in questionnaires using Likert scale items. Out of the p covariates generated, seven had an effect on the outcome. The influential covariates consisted of five normally distributed variables \(X_1, \dots , X_5 \sim \mathcal {N}(0,1)\) and two binary variables \(X_6, X_7 \sim \text {Bin}(1, 0.5)\). The remaining \(p-7\) covariates were normally distributed noise, i.e., \(X_8, \dots , X_p \sim \mathcal {N}(0,1)\). All covariates were simulated as uncorrelated. This design choice was made to limit the scope of the simulation study in which we placed more focus on different effect structures and category response distributions as explained below. For our simulation design, we were loosely inspired by Janitza et al. (2016). However, apart from the normally distributed covariates, we additionally included binary covariates and further included non-linear effects, whereas Janitza et al. (2016) only studied linear effects. We simulated the outcome using the following three different data generating processes (DGPs):The three DGPs were characterized by an increasing amount of non-linear effects. While for DGP 1 all effects were linear, DGP 2 replaced two linear effects by non-linear effects. DGP 3 added an additional quadratic effect. Introducing an increasing amount of non-linear effects allowed for investigating to which point the parametric model still sufficed compared to the TE methods and at which point the TE methods started to outperform the parametric model and should, hence, be preferred. Apart from varying the DGPs, we followed Hornung (2019) in simulating the data according to different class distribution patterns as the author found this to have impacted the methods they have studied. To extend the approach of using different class distributions to more ordinal classification methods as well as to create more diverse scenarios (Hornung, 2019, originally used equally distributed and randomly distributed classes), we included three different class distribution patterns in our simulation. First, a pattern where the classes were distributed approximately equally. Second, a pattern where the middle categories were more populated than the margin categories. Third, a pattern where the margin categories were more populated than the middle categories. Table 1 contains an overview of the relative frequencies per class that we targeted for the different class distribution patterns. They were derived by attaching linear weights to the categories depending on the distribution pattern, e.g., for the pattern “wide middle” with five categories, the weights for categories were 1, 2, 3, 2, and 1. After dividing by the sum of all weights, one arrives at 0.11, 0.22, 0.33, 0.22, and 0.11. The class distributions were obtained by selecting specific values for the thresholds \(\gamma _1, \dots , \gamma _k\) that approximately resulted in the intended relative frequencies for each combination of DGP, number of categories, and class distribution pattern in a dataset of size 100 000. Since this was a heuristic approach, the true class probabilities did not necessarily match the values in Table 1 exactly, hence the term “targeted relative frequencies.” The threshold values chosen for each scenario are listed in Table 3 in Appendix A. Using the simulated linear predictor values and respective thresholds in Eq. 1, the respective cumulative probabilities were computed, transformed into class probabilities, and used for generating class labels from a multinomial distribution.

Fully crossing the settings of DGP, class distribution pattern, number of observations, covariates, and categories resulted in 162 conditions for evaluating the seven methods considered here. We performed 1000 replications, respectively. To evaluate the classification performance, we split the generated dataset into a training set that contained \(\frac{2}{3}\) of the observations for fitting the model and a test set with \(\frac{1}{3}\) of the observations for validating the model. The data partitions were determined by class-stratified sampling. As performance measures, we used the weighted Kappa coefficient \(\kappa _w\) (Cohen, 1968) with linear and quadratic weights as well as Kendall’s \(\tau _B\) (Kendall, 1945) and Spearman’s tie-corrected \(\rho \) (Kendall, 1948). All of these measures are specifically suited for assessing ordinal predictions. Cohen’s weighted Kappa with linear and quadratic weights has frequently been used for evaluating ordinal classification performance (see, e.g., Hornung, 2019; Ben-David, 2008). It is given bywhere \(p^o_{rs}\) is the observed proportion of instances for which r is the true category and s the predicted category, while \(p^c_{rs}\) is the analogous proportion that is expected by chance (Cohen, 1968). The respective weights are denoted by \(w_{rs}\), where for linear weights, \(w^{\text {lin}}_{rs} = 1 - \frac{|r - s|}{k-1}\), and quadratic weights, \(w^{\text {quad}}_{rs} = 1 - \frac{|r - s|^2}{(k-1)^2}\) is chosen. The different weights represent different strategies for penalizing the class distances between predicted and true categories. For linear weights, instances where the predicted categories are equal or close to the true categories are associated with higher weights. For quadratic weights, relatively more weight is attributed to predictions further away from the true category and less weight to predictions close to the true category as compared to linear weights (Hornung, 2019).

To limit the computational burden of the simulation study, we did not perform a hyperparameter tuning for the RF-based methods and used default values instead as RF has been shown to be relatively robust regarding parameter choices (Probst et al., 2019). This decision is in line with previous works from the field (Janitza et al., 2016; Hornung, 2019; Tutz, 2021). For consistency, we have set the number of trees to 500 for all RF methods as this is a common default, e.g., in the ranger and randomForest packages, hence, overriding the default number of trees (for the final forest) in the ordinalForest package that is 5000. In all other cases, the default values remained unchanged.

In the following, the results from the simulation study will be presented. From the 1 134 000 simulation runs (across all 162 conditions, 7 prediction methods, and 1000 replications), we had to exclude 138 runs that led to non-computable correlation values. All of these runs occurred for \(n=250\) when the wide middle class pattern was used. In 135 of these runs, CF was used as the learner, while naive OF (referred to as nOF in the following result figures) was used in the remaining three. The correlation values were not computable due to the model predicting the same class for all observations from the test set resulting in non-existent variability. As such, these runs were excluded from the analysis. Since for all DGPs and performance measures, the findings were consistent across all numbers of categories considered, we are only showing the results for \(k = 5\) categories. The results for \(k = 3\) and \(k = 7\) categories are included in the Online Supplement. For the same reason, we are only showing results for the weighted Kappa with linear weights and Kendall rank correlation scores because the weighted Kappa with quadratic weights and Spearman rank correlation scores led to similar findings, respectively. We also included results for the latter two performance measures in the Online Supplement.

To study whether the results were sensitive to the generative model, we additionally reran parts of the simulation for a generative model that created data according to a linear regression model and binned the outcome into ordinal response categories. As such, this generative model mimicked the approach of transforming numeric outcomes to ordinal outcomes commonly used in practice. To this end, we generated the linear predictor term according to DGP 2 and added standard normal noise. The resulting numeric outcomes were binned such that the targeted relative class frequencies in Table 1 were obtained. This was achieved by approximating the empirical distribution function of the numeric outcome using 100 000 simulated observations. As binning values, the respective quantiles leading to the targeted class distribution pattern were selected. For instance, for the wide middle example mentioned above with \(k = 5\) categories and targeted relative frequencies of 0.11, 0.22, 0.33, 0.22, and 0.11, respectively, we have selected the 11.11%, 33.33%, 66.66%, and 88.88% quantiles as binning values. This approach was inspired by the simulation study in Hornung (2019). Using the alternative generative model for DGP 2, we have obtained results (see Online Supplement) that were consistent with the results described above which indicates that our findings are robust w.r.t. the generative model used for simulation.

Apart from the predictive performance, the required computation time should also be taken into account when comparing the different methods. Figure 4 shows the time needed for training and predicting using a given method relative to the time needed for the CLM. This implies that if absolute runtimes increase overall between conditions, but by the same factor for any given method and the CLM, relative runtimes will be the same in both conditions. Relative runtimes have the advantage of being less machine-specific than the absolute runtimes. The CLM was chosen as the reference method as it represents the classical approach for ordinal prediction as well as the computationally least expensive method. Note that the values have been logarithmized using base 10. As such, a value of 0 means that a given method needed the same time as the CLM, while a value of 2 means that the computation time of a given method was larger by a factor of 100. Since the choices of DGP, number of categories (except for RFSp in which the number of RF fits within RFSp scales linearly with the number of categories), and class pattern had little impact on the relative runtime results, we are only showing the runtime comparison for data generated according to DGP 2 with five categories that were distributed equally. Regarding RFSp, we have decided for five categories as it posed the middle ground of the different numbers of categories considered in our simulation. Since all computations were performed on a compute cluster, it cannot be guaranteed that the individual computations were performed on identical CPUs. Furthermore, we have restricted all methods to only use a single core for computation which may have negatively impacted the runtime of some methods due to not being able to employ parallelization for faster computations. Due to these limitations, our results should be interpreted as broad estimates rather than exact benchmarks. When comparing the relative runtimes, one can see that for all methods, the required runtime was higher by at least a factor of 10 compared to the CLM. From the TE methods, the lowest runtimes were achieved by RF and naive OF which both fit a single RF. As RFSp needed to fit four RF models, it required more runtime than RF and naive OF as expected. In comparison, to the RF-based methods, CF needed relatively higher runtimes for fitting a single model. As expected, the runtimes of OF and OSOA were quite high due to the optimization step. Their computation time was between a 1000 and 10,000 times higher than for the CLM. However, one should keep in mind that the runtime of OF and OSOA is directly linked to the number of optimization iterations, i.e., the number of different score sets evaluated in OF and the maximum number of evaluations in OSOA.

Regarding the impact of n and p, one can see that the relative computation times of the TE methods increase for larger sample sizes. This means that the CLM’s runtimes were less impacted by increased sample sizes than the runtimes of the TE methods. Increasing the number of covariates did not result in a similar effect, indicating that the CLM scaled similarly to the TE methods regarding the computation time.

For practical implications, however, one should also consider the actual runtimes instead of solely relying on the relative runtimes. For example, in the computationally most demanding case of \(n = 2500\) observations and \(p = 35\) covariates, the CLM needed a median time of 0.08 s, RF 3.40 s, naive OF 4.66 s, RFSp 10.26 s, CF 114.93 s, OSOA 394.76 s, and OF 479.79 s. Depending on the dataset and computational power at hand, the discrepancy between, say, a CLM and naive OF or RFSp may be negligible for practical purposes as long as one does not employ a hyperparameter tuning for the RF-based methods.

Similar to the findings in Tutz (2021), our results were also mainly characterized by cases in which the CLM either outperformed the other methods, was on par or notably lagged behind. For the first DGP that included linear effects only, the CLM notably outperformed all ML approaches. This was to be expected as the data generated from a proportional odds model with the all-linear effect structure represented the optimal use case for the CLM. With an increasing amount of non-linear effects, the CLM’s performance suffered in comparison to the TE methods. However, it required quite strong non-linear effects for the CLM to be outperformed. Especially for small sample sizes, the CLM was regularly competitive even in the presence of non-linear effects. For larger sample sizes, the TE learners usually closed the performance gap (when they lagged behind for smaller samples) or widened it (when they were slightly ahead or on par for smaller samples), respectively. Our analysis of real datasets revealed a similar pattern where the most discrepancies between the different methods were caused by the performance of the CLM in relation to the TE methods.

When comparing the TE methods, CF fell behind the RF-based learners once the DGPs increasingly included non-linear effects. This was more prevalent when using the weighted Kappa to assess the predictive performance and less when using rank correlation scores. The differences between the RF-based methods themselves were mostly subtle. The latter finding is in line with Tutz (2021). The performance of our newly proposed OSOA mostly matched the performance of OF. When evaluating all methods on real datasets, we similarly found small differences between the TE methods. The largest performance differences were mostly caused by the CLM that either performed particularly well on a dataset or was outperformed notably by the set of RF-based methods.

Regarding the question of whether optimizing scores in score-based methods such as OF and OSOA nets a benefit over naive OF, our results were mixed. In our simulation, we found that OF improved upon naive OF in cases where the distribution of classes was characterized by dominant middle categories. As such, this supports the findings in Hornung (2019). For the other distribution patterns where classes were distributed equally or a dominant margin categories were present, however, OF could not outperform naive OF. As OSOA’s performance mostly aligned with OF, the findings above also hold when comparing OSOA to naive OF. For the real datasets, OF and OSOA could improve upon naive OF for six out of eight datasets, but the differences were often on a small scale. As such, the benefits of score optimization were rather situational. While it can improve the predictive performance, it is not guaranteed for any given dataset and the high runtimes demonstrated in the runtime comparison must be kept in mind.

While we aimed to make our DGPs as diverse as possible by varying the amount of non-linear effects and the distribution of the response categories, all datasets were generated from models with identical effects across the categories. Future work could study DGPs that include category-specific effects. To this end, it would also be sensible to analyze more parametric models such as partial proportional odds models (see e.g., Brant, 1990; Peterson & Harrell, 1990) or the more recently proposed location-shift model by Tutz and Berger (2022). Additionally, all covariates were simulated as uncorrelated. While it is likely that in individual simulation runs some correlations between covariates randomly occurred, systematically varying the simulated correlation between covariates in future work may help illuminate further differences between the ordinal prediction methods studied here. Furthermore, our work only focused on datasets with relatively few covariates. For high-dimensional data, classical parametric models may run into problems (see, e.g., Zahid & Tutz, 2013). As such, it would be of interest to study how the findings from our comparison translate to high-dimensional settings. Another limitation of our work is posed by the lack of hyperparameter tuning for TE methods. Although this was in line with other works from the field (Janitza et al., 2016; Hornung, 2019; Tutz, 2021) and despite RF’s relative robustness regarding the parameter choice (Probst et al., 2019), a hyperparameter tuning could have amended the performance loss suffered by the TE methods in the simulation scenarios with many noise variables. Further, we did not optimize the parameters of OSOA as its inclusion in the comparison study served more as a showcase. More fine tuning of OSOA’s parameters may yield higher predictive performance or limit the computational burden of the optimization procedure while potentially sacrificing only little predictive performance.

While this work focused on comparing individual prediction methods to assess their respective viability in different data scenarios, Tutz (2021) proposed combining multiple prediction models (e.g., CLM, OF) into a joint ensemble prediction model. To this end, the individual models are trained separately on the training data. For predicting new observations, the predicted response category probabilities of the individual models are aggregated via a weighted mean. The weights are determined in an advance step where the prediction methods are evaluated separately on a subset of the training data and higher predictive performance (relative to the other methods) results in a higher weight. For more details on the joint ensemble approach, we refer to Tutz (2021). When running an ensemble consisting of a CLM, an OF and a RFSp model on the real data examples, Fig. 10 (Appendix C) shows that the joint ensemble achieved the highest linearly weighted Kappa values for two datasets, indicating that this approach can yield benefits. For some datasets, the joint ensemble was partly held back when one of its prediction models performed poorly on the given dataset (e.g., CLM for the Wine Quality data). As such, the joint ensemble approach can be worthwhile to consider, but requires a few design choices whose optimality is likely to vary between datasets, e.g., how many and which prediction methods should be included, what the optimal weighting strategy is, etc. As the joint ensemble fits each prediction method multiple times (due to the weight computation), this approach also leads to extended computational runtimes. Furthermore, if interpretability is of interest, then the ensemble approach loses the interpretability its individual models potentially provide. However, if only the predictive performance is relevant for the application at hand, then joint ensembles can be a sensible approach to employ.

A possible explanation for the lack of consistent improvement upon naive OF could be that OF and OSOA are internally both optimizing a class balanced version of Youden’s J. As Youden’s J is a measure reflecting sensitivity and specificity, it is not a performance measure specifically tailored towards ordinal classification. As such, optimizing Youden’s J may not necessarily result in a more optimal classification result from an ordinal perspective. One could investigate whether optimizing other performance measures such as the weighted Kappa would lead to more consistent improvement upon naive OF.

Specifically for OSOA, the proposed optimization procedure could further be enhanced by including restarts which could help the optimizer in escaping local minima to possibly find better solutions. Additionally, one could try to use multiple sets of starting score values for initializing the optimization instead of only using scores derived from a partition of the [0, 1] interval where all sub-intervals are of equal width. Furthermore, one could explore alternative optimization approaches, e.g., evolutionary algorithms (EAs) such as the covariance matrix adaptation evolution strategy (Hansen & Ostermeier, 1996). EAs keep a population of solution candidates which is continually optimized through recombining and mutating current population members, or sampling new candidates from distributions influenced by current candidates (for an introduction to EAs see e.g., Pétrowski & Ben-Hamida, 2017).

Another possible explanation for the lack of improvement achieved through score optimization may be found in the prediction procedure of OF and OSOA. OF and naive OF determine predictions for new observations by first obtaining the predicted numeric scores from all trees. For each of the tree models, the predicted score is first transformed into a class label by using the associated class borders. The final class prediction is then computed by aggregating over all class labels via majority voting. As OSOA was modeled after OF in this regard, it follows the same prediction procedure. However, since the individual trees in a forest are usually grown without restricting their complexity too much, the resulting terminal nodes may often be pure, i.e., contain (almost) only observations from a single category. Thus, the predicted value for observations falling into the terminal node would simply be the numeric score assigned to the respective category, or a value very close to it in case the node is not entirely pure. Since the predicted numeric score would be immediately transformed into a class label and not processed for further aggregation in its numeric state, the actual numeric score assigned to the respective category may have little impact on the overall prediction behavior. In future work, one could investigate whether score optimization yields greater benefits when first aggregating the numeric scores from the individual trees by averaging and assigning a class to the aggregated prediction score.

Practical machine learning applications often follow a workflow in which the data problem at hand is not approached by only fitting a single prediction method, but instead comparing several prediction methods on a subset of the data to determine which of the many prediction methods available achieves the best performance for the given data. While it would be ideal to compare as many methods as possible, this is often not feasible in practice due to limited time and computational resources. Therefore, one typically resorts to benchmarking only a selected set of prediction methods. Acknowledging this common compromise, we therefore offer the following recommendations that may help arriving at a feasible set of methods to compare for a given data situation. Our work demonstrated that while the RF-based methods can outperform the CLM for non-linear effects and larger sample sizes, it required relatively strong non-linear effects for causing a performance gap. Even in the presence of non-linear effects, the CLM was competitive for small sample sizes. In line with Tutz (2021), we therefore recommend always considering a parametric model as a benchmark model against which other methods should be gauged. For small sample sizes and weak non-linear effects, the parametric model may already suffice and even lead to better performance as the TE methods often require a certain baseline of observations before they can achieve satisfying performance. Having small sample sizes is not uncommon in fields like psychology and social sciences, where ordinal responses are often encountered. For example, in their review of 1568 samples used in psychological publications between 1995 and 2008, Shen et al. (2011) found a median sample size of 172 and a mean sample size of 690 observations. As the TE methods often performed similarly and most differences in our comparison arose from performance gaps between the CLM and the relatively homogeneous set of TE methods, we recommend comparing the parametric benchmark model with a naive OF or RFSp. Classical RF was often lagging slightly behind the other TE methods. The score optimization procedures of OF and OSOA have been shown to be quite computationally demanding and their benefit was mostly situational. Therefore, comparing a parametric model with a computationally less demanding naive OF or RFSp model can give a first indication whether the application at hand is more suited towards parametric models or TE methods. Should naive OF or RFSp outperform the parametric model and should the distribution of the response categories be characterized by dominant middle categories, employing a score optimizing TE method may be worthwhile (if the computational resources permit). Regarding the TE methods, however, one should further keep in mind that despite the RF’s robustness regarding its hyperparameter settings (Probst et al., 2019), scenarios in which one expects a high rate of noisy covariates and a low rate of influential covariates may warrant a hyperparameter tuning. This, in turn, would increase the runtime even further and in combination with a score optimization approach as in OF and OSOA may be less feasible for practical purposes.

Instead of arriving at a prediction model based on the recommendations above, one can also follow the joint ensemble approach from Tutz (2021) with the implications discussed earlier. Since the joint ensemble performs a model selection process internally, it offers a viable alternative to the benchmarking approach. In this case, our recommendations could be used to guide which prediction models to include in the joint ensemble.