Putting ‘political’ back in political trust: an IRT test of the unidimensionality and cross-national equivalence of political trust measures

Despite its widespread use, there is no undisputed understanding of the concept of political trust (Hardin 2000; Warren 2017). Commonly, even if mostly implicitly, political trust is understood to be relational—requiring a subject who trusts (truster) as well as a political object that is trusted (trustee)—as well as situational—applied to specific circumstances or tasks. A trusts B to do X (Hardin 1992). To the extent that trust is an evaluation of this relationship, scholars have proposed various criteria on which these objects are evaluated, such as their competence, motivation, accountability or reliability (van der Meer 2017). Our measures and measurement models of political trust do not cover these theoretical notions very deeply. Political trust is conventionally measured in survey research in a range of objects, such as government, parliament, and political parties (Zmerli and van der Meer 2017). Most measurement thereby incorporates the relational aspect (A trusts B), but not the situational aspect (to do X).

The last decade has seen an increased focus by political trust scholars into the dimensionality of people’s answers to survey items on political trust in a range of political objects.¹ In the factor analytical tradition scholars have focused on the measurement equivalence of trust in various objects (e.g., Breustedt 2018; Marien 2017). This measurement equivalence is a necessary precondition for meaningful comparison (Davidov et al. 2014; Stegmueller 2011). Explaining group differences in political trust only becomes possible if political trust items are understood similarly across these groups. Hence, scholars have looked into the measurement equivalence of scales measuring trust in a range of political objects across time (Poznyak et al. 2014; Smets et al. 2013; Turper and Aarts 2017), country (André 2014; Ariely 2015; Breustedt 2018; Marien 2017; Schneider 2017), time and country (Coromina and Davidov 2013; Marien 2011), subgroups (André 2014; De Vroome et al. 2013; Turper and Aarts 2017), and even levels of analysis (e.g., Ruelens et al. 2018). Measurement equivalence first requires that political trust has a similar structure across groups (configural equivalence), second that the factor loadings—that indicate how well individual trust items relate to this factor—are equal across groups (metric invariance), and ultimately that intercepts have a similar meaning across groups (scalar invariance). In this tradition there is no hierarchical relationship between items themselves. Studies in this field without exception found evidence for metric and even partial scalar invariance. That would imply that items measuring trust in a range of political objects relate similarly to an underlying structure that we label political trust, enabling a meaningful comparison of the underlying structure across time, space and/or subgroups (Breustedt 2018).

A second line of research departed from a different tradition, that of Item Response Theory. More specifically, scholars have tested to what extent political trust items form a hierarchical scale in which trust in one object is a (probabilistic) precondition or precursor to trust in another (e.g., Zmerli and Newton 2017). Mokken scale analysis (Gillespie et al. 1987; Sijtsma et al. 1990) has been the most popular way to test whether political trust items load on this single, underlying, hierarchical political trust dimension, benefiting from differences across items and respondents in means and variances. Mokken scale analysis has been implemented on political trust items in countries such as Italy (Quaranta 2014), the Netherlands (van Elsas 2015), Sweden (Naurin 2011) and the United Kingdom (Rose 2014) as well as in cross-national studies covering Europe (Harteveld et al. 2013; de Vries and van Kersbergen 2007) and the globe (Newton and Zmerli 2011; Zmerli and Newton 2017). All of these studies found evidence that the answer patterns in trust in various political objects indicate strong Mokken scales.² In other words, without exception these studies find strong evidence for an underlying, hierarchical structure that we label political trust.

Despite incidental pushback (e.g. Fischer et al. 2010), there is strong evidence to treat political trust measures as unidimensional and cross-nationally equivalent, as well as evidence for the existence of a hierarchical political trust scale. Yet, various shortcomings and irregularities cast the evidence for cross-national equivalence and unidimensionality in a new light.

First, there is an ongoing question of delineation of political trust by its objects. Some opt for a rather narrow operationalization, focusing primarily on parliament and government as its object (Ariely 2015; Van Erkel and Van Der Meer 2016), whereas others take up a broader definition including a.o. the police and the justice system (Marien 2017), the civil service (e.g., Zmerli and Newton 2017) and international institutions (e.g., André 2014; van Elsas 2015). Few studies actively report specific items that do not meet the demands of the scale (but see Breustedt 2018 who argues that the civil service hinders scalar invariance in her study of cross-national equivalence).

Second, while most studies find evidence for a singular political trust scale or factor, the number of dimensions is not uncontested. On the one hand, Zmerli and Newton (2017) analyzed the World Values Survey and show that political trust can also be understood to belong to an even broader, overarching dimension of societal trust. On the other hand, others show evidence for multiple scales within the subset of political trust items. In her analysis of the World Values Survey data, Breustedt (2018) concludes that political trust is best understood as two factors that she labels representative and implementing.³

Third, most of these scales have been tested in European countries. Only few studies actively incorporate democracies (let alone: non-democracies) outside of Europe. Studies that do encompass a wider set of countries were more likely to reach inconsistent findings. Zmerli and Newton (2017), for instance, find strong Mokken scales of political trust for all democracies in the World Values Survey except India, where they only find a weak scale after eliminating two trust items. Breustedt (2018) finds configural invariance of her two-factor model in only 19 of the 32 countries under study, and full invariance in even fewer.

Fourth, the measurement equivalence literature does not generally find the strongest level of invariance, i.e., full scalar invariance. At best, they tend to find partial scalar invariance (e.g., Marien 2017; Poznyak et al. 2014). Partial scalar invariance is a sufficient condition for meaningful comparisons across groups (Steenkamp and Baumgartner 1998), acknowledging that full scalar variance is in most instances an untenable demand (Byrne et al. 1989). Yet, other studies do not even find partial scalar invariance, particularly across countries (e.g., Ariely 2015; Breustedt 2018).

Fifth, while Mokken analyses consistently find strong, hierarchical political trust scales, the ordering of items from easy to difficult (i.e., most to least trusted) is not constant across countries (Zmerli and Newton 2017). Rather, there are vast differences. In Latin American countries such as Argentina and Chile, government is one of the most trusted political object, whereas particularly in European countries government belongs to the least trusted objects. While political parties are generally the least trusted object, this is not the case in Colombia and Mexico where the civil service is trusted even less. There might be country specific reasons—cultural, institutional, or otherwise—why specific objects are trusted more or less. Yet, although the trust item might itself still be cross-nationally equivalent, too much object-specific cross-national variance would imply that the underlying trust scale is not.

Item Response Theory models strive for a “calibration of both items and persons onto a latent variable scale that represents a construct” (Engelhard Jr. 2008). In other words, in the IRT tradition both the truster (the subject who trusts) and the trustee (the item that is trusted, for practical purposes the trustworthiness of the institution) is positioned on the underlying dimension. In the IRT tradition three main approaches have been put forward: the Guttman, Mokken, and Rasch models (see Engelhard Jr. 2008 for a historical overview). All estimate subject and object positions simultaneously, acknowledging that the difficulty of items might differ. The Mokken and Rasch models are probabilistic, whereas the Guttman model is deterministic. The Mokken and Rasch models differ on various criteria (cf. Engelhard Jr. 2008; Gillespie et al. 1987; van Schuur 2003). Most notably, the Rasch model has stricter demands. Whereas the Mokken scale is a non-parametric test (Sijtsma et al. 1990), the Rasch model is a parametric model with a strict demand of double monotonicity.

A first requirement is monotone homogeneity: subjects who score higher on the latent trait (i.e., political trust) should also be more likely to dominate each individual item (i.e., have a high value on each political trust object). This has several implications. Subjects that dominate a difficult item (a less trustworthy object) should be even more likely to dominate all the easier items (the more trustworthy objects), but not necessarily inversely. An answer pattern in which a respondent trusts the hardest item (the averagely least trusted object) but not the easiest (the most trusted object) constitutes an error. In other words, the easier items function as stepping stones to the more difficult ones. In Mokken scale analysis, monotone homogeneity is a sufficient precondition to measure subject positions (Sijtsma et al. 1990; van Schuur 2003).

Yet, invariant measurement requires double monotonicity: the items that relate to the underlying scale should discriminate equally well albeit at different points of the underlying scale. The difference between ‘easy’ and ‘difficult’ items—reflecting the likelihood of respondents to dominate each item—should be consistent at all points of the scale (van Schuur 2003). Formally, the item response functions should not intercut but rather run parallel. Studies to date have almost without exception focused on the first demand (monotone homogeneity), but not on the more stringent second one.

The Rasch model originates from psychometrics and is rarely used in fields such as behavioral political science. As an IRT approach it stands out by being “grounded in the theory of fundamental measurement” (Annoni and Charron 2019): “First, the calibration of measuring instruments must be independent of those objects that happen to be used for calibration. Second, the measurement of objects must be independent of the instrument that happens to be used for the measuring” (Wright 1968). In this theory of fundamental measurement, measurement is thus not affected by the specific set of be subjects (e.g., respondents) or the specific of objects (e.g., items) to infer the measurement model (Wright and Mok 2004). “An important property of the Rasch model is that, under mild regularity assumptions, consistent item parameter estimates can be obtained from a sample of any subgroup of the population where the model holds. So item parameter estimates obtained using different samples from different subgroups (say, gender or ethnic subgroups) of the population should, apart from random fluctuations, be equal” (Glas and Verhelst 1995).

Double monotonicity is a firm requirement in the Rasch model. Compared to Mokken scale analysis, the assumptions behind the Rasch model are therefore more strict and rigid (Gillespie et al. 1987; van Schuur 2003). It is a parametric model, based on a log transformation of the item and person scores. We can sum up its main assumptions as follows: (1) sum scores are sufficient parameters to calculate the underlying person and item positions on the latent variable of interest; (2) the items constitute a single latent dimension; (3) local stochastic independence: “the response behavior of a person on an arbitrarily selected item g” does not depend on “his or her response on previous items, nor will it affect response behavior on subsequent items” (Meijer et al. 1990); (4) singular monotonicity and double monotonicity.

Figure 1 is a graphical representation of a unidimensional Rasch model of political trust items. Respondents’ likelihood to dominate each of the six trust items (A–F) increases as respondents are more trusting on the underlying political trust dimension. Respondents who dominate the more difficult item are more likely to also dominate all easier items. The ordering of items by difficulty (likelihood to dominate) is constant, regardless of one’s position on the underlying dimension. Finally, all items discriminate equally well, as is evidenced by the parallel trace lines.

There are different Rasch models based on sets of items with varying response category composition. For polytomous items, the most common models are the Rating Scale Model (RSM) and the Partial Credit Rasch Model (PCM) (Andrich 2016). The RSM is developed for a set of items with a constant set of response categories (such as Likert scales) that are also used similarly across items (Ostini and Nering 2006). The PCM has been developed to estimate Rasch parameters for items with varying response categories, and thus has more relaxed constraints (Masters 2016). In this paper we employ the RSM in line with the format of the data. For specific models, however, we also employ the PCM.

The Rasch rating scale model presents the probability to dominate category (k) of item (i) by person (n) as a natural logarithmic function of that person’s position (B_n), the difficulty/trustworthiness of the object (D_i) and the threshold of the category (F_k) (Bond and Fox 2007: 281).

Like most IRT models (Cai and Thissen 2016), there is no single, absolute goodness-of-fit statistic for the Rasch model. Instead, there are multiple tests to assess whether the Rasch model holds. On the one hand, there are overall fit measures such as M₂, a limited-information test statistic that “is found to be especially effective for testing the fit of unidimensional IRT models” (Cai 2016). Model fit may be assessed via traditional parameters such as RMSEA and CFI. On the other hand, there are tests of specific assumptions of the Rasch model. Most notably, assumptions 1 (sufficiency of the sum scores) and 4 (monotone homogeneity and double monotonicity) can be tested by using Andersen’s conditional LR test (Glas and Verhelst 1995: 70; Meijer et al. 1990: 289). The test is based on the premise that subgroups within the dataset have homogeneous item parameters (subgroup homogeneity). It evaluates differences between CML estimates of the item parameters in different subgroups based on score levels or other external criteria.⁴ Assumptions 2 (unidimensionality) and 3 (local stochastic independence) can be tested through the use of the Martin-Löf LR test (Glas and Verhelst 1995: 70, 87). The Martin-Löf test assesses whether two sets of items—i.e., those items that respondents find easy and those that respondents find hard to dominate—tap into the same latent dimensions and thereby meet the demands of the Rasch model.⁵

Up to this point, we performed multiple tests (m₂, Andersen LR, Martin-Löf) in various permutations (RSM/PCM, full range or dichotomized data, with or without the worst fitting items of police and parties, various ways to deal with missing data, bootstrapping) to understand the empirical fit of the very same unidimensional, hierarchical model. The outcomes are not affected by these permutations: The unidimensional Rasch model does not fit the political trust question battery in any democratic country in the World Values Survey (2010–2014). While we do find strong hierarchical (Mokken) scales within most countries, these scales are not at all cross-nationally equivalent.

The strongest robustness check entails fitting the Rasch model on other survey data that tend to differ in wording, number and type of political trust objects, number of answer categories, and most crucially context. Employing a similar setup as in our main analysis (e.g., original range of answer categories; listwise deletion of missing values), we attempted to fit the unidimensional Rasch model on the most recent waves of the Afrobarometer, Arabbarometer, Asian and South Asian barometer, Eurobarometer, European Social Survey, and Latinobarometro. In these robustness tests we did not limit the sample of countries to those with a sufficiently high Polity IV score, but include full, partial and non-democracies. We ended up with 138 national surveys in 119 unique countries and territories in these 6 surveys. The item hierarchies tend to be more similar within these geographic regions/(sub-) continents than across regions, but with the exception of the Arab region they are nevertheless not at all identical.

The Asian and South Asian barometers (2010–2016) covers a rather broad set of trust objects, even though not all objects were questioned in all countries (see the online report for more details). It includes trust in the civil service, the courts, local governments, national government, the national electoral commission, parliament, political parties, the police, and the presidency/Prime Minister. We find weak to strong Mokken scales in all countries, even though the item hierarchies strongly differ across countries. In none of the countries, we find a fitting Rasch model. Some countries (Philippines, Mongolia, Indonesia) have an acceptable RMSEA, but then other fit measures are out of bounds (see appendix Table A3).

The Eurobarometer (wave 87.3 in 2017) stands out because the trust items are dichotomous.¹³ The question battery includes courts, national government, national parliament, the police, the public administration, local government and political parties. Although there are consistently strong to very strong Mokken scales, the Rasch model does not fit the Eurobarometer data (see appendix Table A4).¹⁴ Dichotomous measures might reduce the complexity for respondents, but do not evidently create better fitting Rasch scales.

The European Social Survey (wave 8, 2016–2017) stands out at the other end of the spectrum, for having eleven-point scale trust items, ranging from 0 (no trust at all) to 10 (complete trust). We find consistently strong Mokken scales in each country, and the item hierarchies tend to be rather similar across countries. Moreover, the values of the RMSEA, the TLI and the CFI suggest that in 6 countries the data may adequately fit the Rasch model, i.e., Slovenia, Portugal, Hungary, Spain, France, and Israel (see appendix Table A5). Although stricter and more commonly accepted cut-off values to TLI and CFI (> .90) suggest the Rasch model only fits responses in Slovenia and Hungary, we consider it worthwhile to assess whether the assumptions of the Rasch model are met in all six countries. The Andersen LR test reveals that none of the six countries were close to an acceptable p value for the Andersen LR test. Political trust items with more response categories thus do not necessarily yield better measurement. Rather, these long ranged answer categories may lead to more violations of the sufficiency of the sum-score for person placement on the latent scale.

All in all, the conclusion should be that we find a fitting Rasch model only in 1 of 161 national surveys.