Uncertainty Diagnostics of Binomial Regression Trees for Ordered Rating Data

Learning response patterns from preference and evaluation survey data is a crucial task to be addressed to measure respondents’ perceptions and, subsequently, to understand their behavior. For instance, assessments of churn risk and of customers’ loyalty are usually pursued by asking to rate the extent by which a customer would recommend a given service or product to others. Similar investigations concern the assessment of perceptions and opinions about public policies and interventions on relevant subjects. For these data, the binomial regression model is a well-acknowledged choice since it offers a simple and effective representation of the data generating process (Allik, 2014; Grilli et al., 2015; Pinto da Costa JF. et al., 2008; Raidvee et al., 2012; Zhou & Lange, 2009). The estimable probability parameter conveys a synthesis of the latent feeling with versatile interpretation (satisfaction, preference, agreement, etc., depending on the topic under investigation). Regression techniques and standard variable selection algorithms allow to derive relevant response profiles in terms of subjects’ characteristics. Automatic identification of these response profiles can be achieved via tree methods: in particular, the rationale of model-based binary trees (Zeileis et al., 2008) hinges on the assumption that a given model can be maintained at each partitioning level, thus performing an iterative search for significant splitting variables both to disclose variables interaction and to optimize model fit, on the basis of parameter instability tests. In this framework, the paper discusses the implementation of binomial regression trees for rating data. In order to account for possible framing effects on the response outcome (Hilbig, 2012), the class of CUB mixture models and their extensions (Piccolo & Simone, 2019) provides a research paradigm to assess both feeling and uncertainty of the response distribution in a parsimonious way (Tourangeau et al., 2000). Indeed, any measure of location and shape summarizing the overall feeling should be accompanied by an assessment of the heterogeneity of the distribution and should be adjusted for possible nuisance effects blurring the underlying sentiment, as those induced by inflated frequencies and over-dispersion, for instance. Under this rationale, if the binomial model is assumed for the feeling component, different uncertainty specifications can be considered: the uniform distribution for the so-called non-contingent response style, or specific models to cope with different response styles (Gottard et al., 2016; Simone & Tutz, 2018). As for every Mixture of Experts System (Gormley & Frühwirth-Schnatter, 2019), variable selection for regression of the featuring mixture parameters is a challenging task. Recently, model-based trees have been developed for the class of CUB models (Cappelli et al., 2019; Simone et al., 2019), providing useful and innovative tools to perform automatic variable selection and to describe and classify response profiles in terms of uncertainty, feeling, and possible shelter effect, occurring where an excess of frequency is observed at a category acting as refuge response. Motivated by the circumstance that the assumed model might not provide an adequate fit to all response profiles, the paper wishes to contribute to the state of the art addressing the issue of model diagnostics on the response profiles learnt from a model-based tree, with focus on the binomial case. Specifically, surrogate residual analysis proposed in Liu and Zhang (2018) is exploited to run a local model selection for the best binomial extension that verifies a necessary condition for being correctly specified, within the class of mixture models with uncertainty.

The paper is organized as follows: binomial trees for ordered evaluation data are discussed in Section 2. Section 3 recalls some recent findings on residuals’ analysis for ordinal regression models and provides an overview on how this approach can be exploited as a diagnostic procedure for the chosen class of models and related trees. The ideas put forth in the paper are illustrated in Section 4 on the basis of a customer satisfaction survey as an introductory example. Then, Sections 5 and 6 present two more comprehensive case studies for the analysis of perceived trust towards Television and Press — taken from the ALLBUS German General Social Survey (GESIS - Leibniz-Institut für Sozialwissenschaften, 2016) — and for the analysis of satisfaction of Italian Ph.Ds. for the doctoral studies, taken from the ISTAT survey on carrier placement. ¹ An outline of future developments is given in Section 7.

Throughout the paper, assume that ordered evaluations are collected on a discrete response scale with m ordered categories, say c₁ ≺ c₂ ≺⋯ ≺ c_m: categories will be coded as integers for notational convenience only (c_j = j). Let R denote the response rating variable and assume m > 4 for identifiability of all the models that will be considered.

In order to parameterize the latent sentiment and possible framing effects for the response R, the setting of models on the discrete support is advocated. The (shifted) binomial distribution: provides a parsimonious yet effective choice for the response R, acknowledged by several literature (Allik, 2014; Pinto da Costa JF. et al., 2008; Raidvee et al., 2012; Zhou & Lange, 2009). The estimable parameter ξ ∈ (0,1) accounts for the location and shape of the response: specifically, parameter 1 − ξ assesses the probability that a category is preferred over the previous ones. ², so that the highest is the value of ξ, the more the distribution will be right-skewed with modal value at one of the lower categories. Thus, if the response scale is oriented so that higher categories correspond to a stronger trait, the measure 1 − ξ is a direct indicator of the overall feeling towards the item being investigated.

Adhering to the rationale supported by some experimental psychologists (Tourangeau et al., 2000), the response process of ordered evaluation data on a discrete support is deemed to be subject to framing effects (Hilbig, 2012). The resulting conceptualization achieved via the class of mixture models with uncertainty (Piccolo & Simone, 2019) involves a combination of actual feeling towards the item and uncertainty, so that: where \(\mathbf {q} = (q_{1},\dots ,q_{m})\) is a suitable model for uncertainty contaminating the feeling model, and π ∈ (0,1] is the mixture parameter weighting feeling importance. The benchmark choice is the uniform specification encompassed by CUB models (\(q_{r}=\frac {1}{m}\) for all \(r=1,\dots ,m\)), to account for heterogeneity of responses and adjust the feeling assessment accordingly (for short, \(R \sim \) CUB(π,ξ)). In several circumstances, a category acts as a refuge option for the response choice, resulting in an inflated frequency for that category, so that uncertainty degenerates solely or partly to a shelter effect (Iannario, 2012). In this case, the feeling measurement should be corrected accordingly, by letting the contaminating distribution in (2) be: with \(\text {D}_{r}^{(c)}\) denoting a degenerate distribution with mass concentrated at the shelter category, say c, and δ ∈ [0,1] measures the excess frequency at c. The resulting distribution for the observed response is a CUB with shelter effect at category c, namely: with shelter effect parameterized by δ = 1 − π₁ − π₂, π₁,π₂ ∈ (0,1) weighting the importance of feeling component and heterogeneity, respectively. This model encompasses the binomial with shelter (π₂ = 0). Further uncertainty specifications are possible to encompass different alleged framing effects (Gottard et al., 2016), as well as to consider response styles to middle and extreme categories by choosing the discretized beta distribution (Ursino & Gasparini, 2018; Simone, 2022) as contaminating model (resulting into CAUB: Combination of adjusted uncertainty and Binomial, see Simone and Tutz (2018)). In general, any discrete distribution with support in \(\{1,2,\dots , m\}\) (or a discretized version of a continuous distribution) with some interpretative extent as nuisance of feeling, can be assumed for the uncertainty component, provided that it does not entail identifiability problem when mixed with the feeling component.

so that the binomial is recovered as \(\phi \rightarrow 0\). As a result, mixtures of the beta-binomial model with (discrete) uniform distribution or with degenerate distribution at a fixed category can be designed to account for heterogeneity (resulting into CUBE models: Iannario (2013)) and for shelter effect (beta-binomial with shelter).

Given this choice, the implementation of model-based trees (Zeileis et al., 2008) is a natural step to characterize response profiles with synthetic information of different response features (feeling, heterogeneity, shelter, over-dispersion), thus fostering their interpretation and comparative analysis.

For models of the form \(R \sim F_{a}(r; X, \theta )\), where F_a(⋅) is the cumulative distribution function of the assumed model, Liu and Zhang (2018) advocate a jittering approach on the probability scale to define residuals. Briefly, a surrogate variable S is defined by conditionally sampling from a continuous uniform distribution over (0,1): On this basis, the residual variable V for the fit of the assumed model can be defined as: where η denotes the available information set. One of the main findings of the results established in (Liu & Zhang, 2018, Section 7, Theorem 4) is that \(V| X \sim \mathcal {U}(-\frac {1}{2},\frac {1}{2})\), if the assumed model is correctly specified. Thus, if this necessary condition for the assumed model does not hold, mis-specification is detected. The proposal of the paper is to resort to diagnostics of residuals’ built via the surrogate variable method to determine if the assumed model can be maintained or if it is misspecified in either some model components or in neglecting mixed populations. Beyond graphical inspection of residuals, general tests for comparisons of continuous distributions (as the Kolmogorov-Smirnov, Cramer-von Mises tests), as well as specific tests of uniformity, can be considered to perform residual diagnostics. In the following, the Quesenberry-Miller test of uniformity (Quesenberry & Miller, 1977) will be considered to test if an assumed model verifies the necessary condition for being correctly specified. This choice is due to its best power performance against alternative tests (Quesenberry & Miller, 1977).

In the next subsections, we show how to adopt this method to binomial regression trees to perform local uncertainty diagnostics. Specifically, the ultimate goal of the paper is to exploit this necessary condition to determine if an assumed model (hereafter, the binomial) can be maintained at each step of the partitioning process, or if a split should be preferably pruned. In the first case, local model selection for the best binomial extension that fulfils the necessary condition for being correctly specified can be pursued to improve the interpretative extent of the tree.

The ABC Annual Customer Satisfaction Survey refers to a company offering IT solutions to media and telecommunication service providers (Kenett & Salini, 2012, Chapter 2). On a Likert-type scale with m = 5 ordered categories (1 =“very low,” 5 =“very high”), customers were asked to rate their overall satisfaction, along with satisfaction for several aspects of the customer experience, including the following: the extent by which they would recommend the ABC company to a third company (recom); the extent by which the would reconsider the company for further purchases (product); the overall satisfaction for the equipments of the purchase (equipment); for sales and technical support (sales, technical); purchasing support (purchase) and pricing. Due to the small sample size obtained after omitting missing values list-wise (n = 212), only small trees can be grown: thus, this dataset will be used for illustration purposes only.

Table 3 reports the average p-value for the Quesenberry-Miller test of uniformity for surrogate residuals for selected candidate models (50 random generations were considered), showing that — at the 5% level — the binomial is significantly misspecified for ratings concerning recom, equipment, technical. For equipment, the only model that fulfils the necessary condition for correct specification is the binomial with the addition of a shelter effect (at category c = 4).

It is seen that the binomial model (without covariates) cannot be maintained for recom at the fixed significance level. This circumstance may be due to missing sub-populations: indeed, the primary split of a dissimilarity binomial tree separates recom ratings provided by customers who are not satisfied with the sales support (sales≤ 2) from those who are satisfied (sales≥ 3), for which the binomial model can be safely assumed instead. Figure 5 displays diagnostics check of uniformity of residuals at the root node and at left and right descendants (top row panels), along with the barplots of the frequency distributions at the nodes, with superimposition of the fitted Binomial model.

Node 2 is declared terminal by the procedure due to a pre-pruning condition relative to the sample size of the node: the split that is selected for node 3, instead, cannot be accepted for the binomial tree since the (conditional) binomial is significantly misspecified for its left descendant (node 6: see Fig. 6). Thus, the tree growing procedure stops and node 3 is declared terminal as well.

Thus, one should prune this further split under the binomial tree.

Table 4 reports the average p-value obtained over 50 replications of residual generation for competing models: it is seen that the binomial and all the extensions satisfy the necessary condition for being correctly specified at the given significance level. Thus, local model selection can be performed to identify the best uncertainty specification according to standard criteria.

Accordingly, Table 5 summarizes main information for each node: estimated feeling measure \(1-\hat {\xi }\), mixing weight \(\hat {\pi }\) of the feeling component.⁴ possible shelter effect parameter \(\hat {\delta }\) and corresponding shelter category under the best mixture model with uncertainty, as well as dissimilarity of both binomial and best model with respect to the observed frequency distribution. For each inner node of the tree, the selected split variable and split point to determine the left and right descendants are also reported. The best model is selected by jointly considering results from likelihood ratio tests for nested models (in particular, with respect to the baseline binomial), and from BIC comparisons for non-nested models. In case more models are equivalent in terms of BIC index (Burnham & Anderson, 2003), the model with the lowest dissimilarity with the observed frequencies can be chosen.

From response profiles learnt at terminal nodes (3,5,8,9), it can be claimed that:

Perceived quality of products and services is related to perceived trust of users in a complex and multi-faceted scheme that involves customers’ satisfaction and loyalty (Bloemer et al., 1999; Chiou & Droge, 2006; Eisingerich & Bell, 2008; Garbarino & Johnson, 1999).

With reference to the ALLBUS German General Social Survey of 2012 (GESIS - Leibniz-Institut für Sozialwissenschaften, 2016), consider the perceived trust expressed by n = 2692 respondents towards Press and Television, as institutions, collected on a rating scale with m = 7 ordered categories (1 = “no trust at all,” 7 = “a great deal of trust”), after list-wise omission of missing values of the considered set of variables.

Among the available covariates used to grow the tree (including gender, employment status, German citizenship, income, and marital status), the procedure has selected:

For both ratings on perceived Trust towards Press and Television, the response profiles derived from the dissimilarity binomial tree are discussed. Indeed, for latent traits like perceived trust which are deemed to exhibit similar sentiment among the population, the dissimilarity criterion can be helpful in determining significant differences in model parameters that highlights more dissimilar response patterns. Given the orientation of the response scale, the feeling measure \(1-\hat {\xi }\) under the binomial model is a direct indicator of perceived trust.

The approach proposed in the paper takes the lead from the introduction of the binomial tree, since it is the simplest model on the discrete support that can be assumed for evaluation data to parameterize the latent feeling. However, the diagnostic procedure and the subsequent local specification adjustments can be implemented for every model-based tree with respect to extension of the assumed model. For illustration purposes, an example on CUBREMOT is presented (Cappelli et al., 2019; Simone et al., 2019). Consider the overall satisfaction for the doctoral experience rated by Italian Ph.Ds. awarding the title in 2012 and 2014, collected within the survey run by the Italian National Statistical Office (ISTAT) to investigate their satisfaction for the professional placement and the Ph.D. programme (available at https://​www.​istat.​it/​it/​archivio/​87536).

Satisfaction for the Ph.D. experience was rated with reference to several dimensions (quality of teaching courses, spaces and tools at disposal, and so on): ratings for overall satisfaction will be considered hereafter as response variable. All the ratings were collected on a discrete scale with categories coded from 0 to 10: the rating scale has been subsequently modified to a scale with 8 categories due to zero-scores observed in certain categories, so that higher scores along the response scale corresponds to higher levels of satisfaction.

After omitting missing values for the variables of interest, n = 3830 observations are used for the analysis. Among the available covariates used to grow the tree (including current employment status, residence, discipline of the Ph.D. program, marital status, participation in research project, and year of Ph.D. completion), the procedure has selected:

Due to a structural heterogeneity of the distribution, the binomial does not satisfy the necessary condition for being correctly specified at any of the partitioning levels for the corresponding model-based tree, not even after searching for possible neglected sub-populations. Indeed, for a general CUB model, the necessary condition for being correctly specified is verified at each stage of the growth of a CUBREMOT with the deviance splitting criterion (results are not reported for brevity). Thus, CUB can be assumed as maintained model for the response generating process. Then, at each step, the procedure selects the most significant binary split in either feeling or uncertainty parameters. Table 14 reports the main information concerning the nodes composing the tree with respect to the local search of the best model extension, among those verifying the necessary condition for being correctly specified (in particular, candidate models are CUBE and CUB with shelter) see Fig. 11 for visualization of results.

First, it is worth to highlight that a CUBREMOT tree with a pre-specified shelter fixed at all partitioning levels would have entailed sub-optimal descriptions of the response profiles. Once diagnostic checks have confirmed that the baseline CUB can be maintained as assumed model, the flexible model selection that is performed locally provides, instead, more accurate descriptions of the response patterns (for instance, shelter effects are found at different levels of the scale for different response profiles). In particular, it follows that:

Statistical modelling of preference and evaluation data that accounts for uncertainty can be embedded in the 7th dimension of Information Quality (Kenett & Shmueli, 2014; Kenett, 2016), a paradigm for analytic research that guides scholars and stakeholders towards a qualified experience of data analysis to successfully pursue research goals and decision-making process. Being framed within this modern debate, the paper proposes a method to perform local diagnostics of model-based trees for evaluation data, assuming the setting of mixture models with uncertainty. The discussion stems from the introduction of binomial regression trees, following the rationale of CUBREMOT technique (Cappelli et al., 2019; Simone et al., 2019). With respect to the consolidated approach to model-based trees, which assumes a constant model specification at all partitioning levels, the local-model selection for the best extension of the binomial model at tree nodes allows for a more accurate assessment of feeling. The adoption of this flexible approach is subject to the fulfilment of a necessary condition for correct specification that can be assessed through diagnostics of surrogate residuals for ordinal data models. Then, a pruning criterion could be derived for those splits where evidence for model misspecification is found. In a similar way, diagnostics check for model-based trees could be exploited to tune the depth of the tree, or to select the best tree out of a set of alternative ones. The thread of the discussion has been the binomial model for rating data, but the proposal may be adapted to other preference models on the discrete support (as shown in Section 6 for the case of CUBREMOT). In the end, the flexible binomial tree with uncertainty could be possibly exploited to obtain: a more precise imputation of missing values or prediction rules on the basis of the derived response profiles and its characterizing model parameters; an adjusted synthetic indicator of the latent feeling of the trait being examined. Further research will address the possibility of incorporating the proposed diagnostics methods within the partitioning process to drive the selection of the split and provide more general flexible uncertainty trees: see Banchelli (2019) for a recent proposal of flexible trees on the basis of non-nested models for count data. With respect to residuals diagnostics, further methodological research will assume a comparative perspective with alternative tests and techniques to detect model misspecification; adjustments of the splitting rule on the basis of stability tests of regression parameters, as for the model-based trees introduced in Zeileis et al. (2008); a sensitivity analysis to determine the most suited synthetic indicator of the distribution of the p-values for the uniformity tests for residuals. Finally, the proposal could be exploited for prediction purposes with the implementation of model-based random forests for rating variables. In this setting, specific attention should be devoted in the derivation of variable importance measures due to possible indirect to effects of splitting variables (Gottard et al., 2020), which could also affect the interpretation of response profiles read from a single tree.