Measuring Polyarchy Across the Globe, 1900–2017

This component index measures whether the legislature and the chief executive are elected—the latter either directly through popular elections or indirectly through a popularly elected legislature appointing the executive. This is sometimes referred to as the “effectiveness” (Hadenius 1992, p. 49) or “decisiveness” (O’Donnell 2001, p. 13) of elections. A “popular election” is minimally defined and includes sham elections with limited suffrage and no competition, and “appointment” by the legislature only implies selection and/or approval, not the power to dismiss.⁸ Unlike the other four component indices, this index is based on a set of formative (or constitutive) indicators that define the construct on purely theoretical grounds. The logic behind the construct is schematically portrayed in Fig. 2.

There are six links of appointment/selection of the chief executive to account for: first and second, whether the head of state (a1) and/or head of government (a2) is directly elected (1) or not (0); third, the extent to which the legislature is popularly elected (b), measured as the proportion of legislators who are directly or indirectly elected (if the legislature is unicameral), or if the legislature is bicameral and the upper house is involved in the appointment of the chief executive, a weighted average of the proportion elected for each house;⁹ fourth and fifth, whether the head of state (c1) and/or head of government (c2) is appointed by the legislature (1), or the approval of the legislature is necessary for the appointment of the head of state (1), or not (0); and sixth, whether the head of government is appointed by the head of state (d = 1) or not (d = 0) (see Appendix B on exact question wordings and response categories).

In polities with unified executives (that is, when the head of state is also the head of government, Elgie 1998; Siaroff 2003), the complexity of this conceptual scheme reduces to the links a1 and b*c. Since these are considered perfect substitutes (either a directly elected president or a president elected by a directly elected parliament suffices), the index value is arrived at by taking the maximum value of the two. In dual systems, where there is both a head of state and a separate head of government, the chief executive is determined by comparing the two executives’ power over the appointment and dismissal of cabinet ministers. If the head of state and head of government share equal powers over the appointment and dismissal of cabinet ministers, the index averages across the degree to which they are directly or indirectly elected. The resulting construct is called the elected executive index (v2x_accex).

In the second step, the extent to which the legislature is elected (b) is also independently taken into account in order to penalize presidential systems with unelected legislatures, or legislatures with a large share of presidential appointees, for example. Assuming that the extent to which the chief executive and the legislature are elected work as partial substitutes, we average the two. The resulting index of Elected Officials (called v2x_elecoff) has a bimodal distribution (see Appendix Figure D1). The rare values falling between 1 and 0 are mostly bicameral systems where the upper house is indirectly elected, but also include cases such as Burma/Myanmar that (currently) has a unified executive with a president elected by a parliament in which only 75% of the seats are directly elected.¹⁰

The other polyarchy components are arguably better measured by reflective indicators that partially substitute for one another. With the exception of “Inclusive Citizenship,” which we capture by the extension of the suffrage,¹¹ we include multiple indicators for each component to allow both for a more comprehensive appraisal of each component, and for assessments of the extent to which they load on a single dimension. To test the latter proposition, we run a Bayesian factor analysis (BFA) model for each of the three components “Free and Fair Elections,” “Associational Autonomy,” and “Freedom of Expression and Alternative Sources of Information.” The results are reported in Appendix Tables A1–3. In each case, there is clear evidence for unidimensionality. We therefore construct a BFA index for each of these components in turn, accompanied by measures of uncertainty.¹²

The first, which Dahl (1998, p. 85) calls “free, fair, and frequent” elections, but which we designate “Clean Elections Index” (v2xel_frefair), is designed to capture the level of integrity of elections (e.g., Schedler 2002; Lehoucq 2003; Birch 2011; Kelley 2013; van Ham and Lindberg 2016). Eight indicators constitute this index, measuring the (a) autonomy and (b) capacity of the election administration body (EMB) to conduct well-run elections; and for each election, the extent of (c) registration irregularities, (d) vote buying, (e) ballot fraud and intentional irregularities, (f) government-induced intimidation of opposition candidates, (g) other types of election violence (not instigated by the government or ruling party), and (h) an overall assessment of whether the election overall was “free and fair.”¹³ The resulting index is bimodally distributed but with a skew towards observations with no elections (see Appendix Figure D4).¹⁴

The second component captures what Dahl (1998, p. 85) calls “Associational Autonomy” (or freedom of organization) using six indicators. The core of this construct is party-centered: are political parties free to form and operate, and are opposition parties allowed autonomy from the ruling regime? Equally important, and implying that there is an electoral aspect also to this component: are parties free to field candidates in national elections making them truly multiparty? Yet, Associational Autonomy in the political sphere additionally requires that there be no state repression, or barriers to the entry and exit, nor repression of a wider set of civil society organizations providing alternative means for voice and political activity. The resulting index (v2x_frassoc_thick), rescaled to 0–1 through the normal cumulative distribution function (cdf), is bimodally distributed (see Appendix Figure D5).¹⁵

Finally, Dahl’s concept of polyarchy includes two conspicuously non-electoral aspects: freedom of expression and access to alternative sources of information. Staying true to his widely accepted concept thus necessitates employing a core set of indicators capturing overall media freedom (Behmer 2009), such as active state censorship of print/broadcast media, media self-censorship, and harassment of journalists. In addition, it requires freedom of discussion in society at large, for both men and women (Skaaning 2009), and four indicators of media content to capture Dahl’s (1971, 1989, 1998) “alternative sources of information”: whether the media is biased against opposition parties and candidates, whether major print and broadcast outlets routinely criticize the government, and whether they represent a wide range of political perspectives, as well as general repression of cultural and academic expressions of political dissent. As argued above, one could question whether these really belong to a mostly electoral conception of democracy. We contend that Dahl (1989, 1998) himself provided the most viable answer to that question through his treatment of the notion of “enlightened understanding.” In essence, elections may be free and fair, and suffrage universal, yet without these non-electoral institutional guarantees in place, elections would still be meaningless if voters do not make an informed choice based on at least some minimal possibilities for collective deliberation. We therefore include both these guarantees using nine indicators but collapse them into one index called Freedom of Expression and Alternative Sources of Information (v2x_freexp_altinf) since they are very strongly correlated (results in Appendix Table A3). This is again converted to a probability (0–1) score by using the normal cdf, resulting in the distribution in Appendix Figure D7.

Assuming that V-Dem experts exercise independent judgment when translating their perceptions of the world into numerical ratings (Schedler 2012), coder agreement could be interpreted as a sign of validity (Steenbergen and Marks 2007, p. 351). Table 3 decomposes the variance in the 23 expert-coded indicators (out of the 40 in total) underlying the polyarchy index: Clean Elections (8), Associational Autonomy (6), and Freedom of Expression and Alternative Sources of Information (9).¹⁹ Results are pooled across indicators for each component, controlling for country- and year-fixed effects. With random effects measured on the same ordinal 0–4 scale as the original indicators to ease interpretation, the standard deviations are thus only about a half point on a four-point scale suggesting very modest levels of disagreement among the country experts. One should also recall that a decent share of this “raw” coder disagreement is known to be a reflection of varying coder thresholds for mapping perceptions onto the ordinal coding categories, and that the measurement model corrects a substantial share of this differential item functioning when aggregating the raw scores to the country-year level (Pemstein et al. 2018).

Granted that country experts disagree at times, are there predictable patterns in that incongruity? Arguably, we should expect coders to disagree more, the more difficult the coding task (Steenbergen and Marks 2007; Coma and van Ham 2015). In V-Dem data, we may distinguish at least three sources of difficulty (McMann et al. 2016). First, all else equal, judging indicators of electoral democracy should be more difficult further back in time, when experts cannot rely as much on their academic and lived experience. Second, difficulty should increase when there is little independent information from which to form one’s judgment. Third, extreme conditions should be more easily identifiable than the “muddled middle”; hence, coders should be more likely to disagree at intermediate levels of whatever indicator they are assessing. This third expectation may work counter to the first to the extent that the situation in many countries was quite undemocratic aspects in the early years of the twentieth century.

In Table 4, we test these three propositions by regressing the country-year standard deviation in coder ratings, pooled across the same 23 expert-coded indicators as in Table 2, on year of observation, an independent measure of media access,²⁰ the level of the aggregate coder rating (a product of the measurement model), and that level squared. We present results by component, controlling for indicator-fixed effects (not displayed in the table). We also control for the number of coders, which may in itself be a systematic driver of the amount of disagreement.²¹ All three expectations are borne out by evidence. Coders disagree more when coding electoral democracy indicators further back in time, when there is less available media information, and at intermediate levels of the indicators they are assessing. There are only two exceptions. First, for Associational Autonomy, the largest degree of coder disagreement is actually at the lowest levels; second, the estimate of observation year for the clean elections indicators is insignificant (implying that election quality is equally difficult to assess across time when holding the amount of available information constant). We conclude that by and large, the level of coder disagreement varies in a meaningful and predictable way, lending further support to the validity of V-Dem data.

As a final test comparing coder estimates, we assess whether coder characteristics from the V-Dem post-survey questionnaire systematically affect ratings following Dahlström et al. (2012). We control for gender (female = 1, male = 0); age (in years, also including the squared polynomial); level of education (PhD = 1, 0 = less than PhD); employment (1 = government, 0 = other); whether the expert was born in or resides in the country he or she is coding (1 = yes, 0 = no); and whether the expert is of “Western” origin or not. Additionally, we control for three potential sources of ideological bias: support for free market economy as a proxy for left-right leaning, support for two different normative understandings of democracy. The first we call the “conventional understanding,” which entails support for electoral and liberal democracy. The second we call the “alternative understanding,” which entails support for participatory, deliberative, or egalitarian types of democracy.²²Appendix F provides the exact question wording for all the above. In order to fix comparisons across coders to the same countries and years, we also control for country- and year-fixed effects (results omitted).

The results in models 1, 3, and 5 of Table 5 are comforting overall. With two exceptions, characteristics are not significantly related to how V-Dem experts rate electoral democracy indicators. The exceptions are (a) the tendency among coders born in the country to provide a more positive assessment of election quality than non-native experts and (b) those supportive of “alternative” understandings of democracy to provide lower ratings on freedom of expression than “conventionalists.” These are interesting patterns worthy of further inquiry, but they are the exceptions to the rule: experts tend to provide similar assessments of election quality, freedom of association or of expression, independent of who they are.

Models 2, 4, and 6 test for another type of coder bias that Bollen and Paxton (2000, p. 72) label “situational closeness.” This is the idea that “judges will be influenced by how situationally and personally similar a country is to them. “Accordingly, strong believers in the free market would have a specific tendency to rate countries with free markets better. Similarly, situational closeness bias implies that believers in other conceptions of democracy rank countries more proximate to their ideals more favorably also on the electoral indicators. Finally, this kind of bias could manifest in such a way that even if Western coders displayed no systematic bias for their countries, bias might be directed towards non-Western countries. We test this by controlling for interaction effects between particular coder and country characteristics.²³ None of these interaction effects are, however, significant at conventional levels. The dominant pattern is thus that situational closeness does not introduce bias in the V-Dem experts’ ratings of electoral democracy.

A second set of validation exercises focus on comparisons between our measure of electoral democracy and extant alternatives. Figure 5 displays the bivariate descriptive pattern at country-year level, comparing the V-Dem polyarchy index to the Polity and Freedom House ratings, as well as the “Unified Democracy Scores” (UDS). The latter is based on just about every pertinent measure of electoral democracy (except V-Dem) covering multiple countries and years (Pemstein et al. 2010). Convergence is the overall pattern. The pairwise correlations range from .85 for Polity to .94 for the UDS ratings. The red smoothed lowess-line of best fit also indicates a consistent pattern of monotonically increasing levels of polyarchy, the higher the extant measure of electoral democracy. Without treating any of the particular measures as gold standards, in the convergent validity sense, these different measures validate one another.

Nonetheless, the polyarchy index regularly produces scores that differ from extant measures. Some pertinent examples of this were highlighted already in Fig. 1, where we argued that the V-Dem measure outperforms extant measures in terms of face validity. In Figs. 6 and 7, we portray the country differences more systematically by plotting the average polyarchy scores against the average Polity and FH scores, taking into account every country year that exists in both respective datasets.²⁴ The diagonal indicates where there is no difference between the measures, implying that V-Dem polyarchy rates electoral democracy higher than the alternative measures for cases above the line, but lower for the ones below the line. There are a few cases above the diagonal (the top five are highlighted in red). Three cases appear on this list for both Polity and Freedom House: the German Democratic Republic (GDR), Turkmenistan, and Uzbekistan. The most important reason that V-Dem tends to score those countries higher than both Polity and Freedom House is that Elected Officials and Suffrage, which both score high in these cases, are allowed to exert an independent influence on the polyarchy scores.

The bulk of cases, however, appear below the line (the average difference with Polity is − .13, with FH − .06), meaning that the polyarchy index is a more demanding standard on average than the alternative measures. Several of the most diverging cases for Polity are presented and discussed already in Fig. 1. (Colombia ranks 12th in terms of how far it falls below the diagonal, so it is not highlighted in Fig. 6.) But for Freedom House, there is an additional conspicuous pattern: several of the countries where FH is significantly larger than polyarchy are relatively small states, such as Barbados, the Dominican Republic, Montenegro, Papua New Guinea, and the Solomon Islands. What drags down these countries in particular in the polyarchy index is the low quality of their elections as compared to their FH ratings.

In sum, the polyarchy index is positively correlated with extant measures of democracy, but we also find discrepancies where we believe polyarchy, by incorporating all of Dahl’s five components and by relying on more fine-grained case expertise, provides a more nuanced and accurate picture of the state of electoral democracy.²⁵