Could vote buying be socially desirable? Exploratory analyses of a ‘failed’ list experiment

Decades of survey research has established that individuals often do not provide truthful answers when asked sensitive questions (Blair et al. 2020; Glynn 2013; Tourangeau and Yan 2007; Fisher 1993; Rinken et al. 2021). This occurs across diverse types of sensitive topics, from drug-use to voting behavior, and is a problem recognized in nearly all social science literatures (Fisher 1993).

The problem, known as sensitivity bias,¹ has been a significant challenge in the study of vote buying. Vote buying is a particular form of clientelism which involves the exchange of money or material incentives in return for votes (Gonzalez-Ocantos et al. 2012; Stokes 2007). In many countries, vote buying is both illegal and socially stigmatised (Stokes 2007), making it a sensitive topic to ask individuals about. By now, scholars take the under-reporting of vote buying as given, and few scholars would ask about vote buying without using a technique for the measurement of sensitive topics. In particular, many scholars have turned to list experiments– a measurement technique which offers surveys respondents confidentiality in order to encourage truthful reporting.

Following common practices, we employed a list experiment within a nationally representative survey in order to measure vote buying in the 2019 Nigerian general elections.² We also measured vote buying using a direct survey question. When comparing the list experiment estimate of vote buying and the direct measure, we find that the reported prevalence of vote buying is 22 percentage points higher in the direct survey question. Often this result –the over-reporting of a sensitive behavior³– is viewed as a list experiment “failure” (e.g., Kramon and Weghorst 2019). This kind of result is not uncommon, occurring in contexts as diverse as shoplifting in Japan (Tsuchiya et al. 2007) and support for same-sex marriage in the U.S. (Lax et al. 2016). Yet, the over-reporting of a sensitive behavior is not well understood, and has contributed to a file-drawer problem in the context of list experiments (Blair et al. 2020; Kramon and Weghorst 2019; Gelman 2014; Castro Cornejo and Beltrán 2020, 2022).

In our exploratory analysis, we show how assumptions about polarity (what the sensitive response is and whether under- or over-reporting is expected) affect the conclusions we draw from a list experiment.⁴ In particular, we show how the interpretation of tests meant to diagnose design failure also depend on polarity assumptions. First, following standard methodology, we find no evidence of failure due to violations of the “no design effects” and “no liars” assumptions of list experiments (Blair and Imai 2012). Respondents do not appear to adjust their responses to the list experiment depending on the inclusion of the vote buying item (no design effects). Formal tests for “floor”- and “ceiling”-liars suggest that respondents do not conceal vote buying in the list experiment. However, we find evidence that respondents intentionally reveal vote buying in the list experiment, which could be interpreted as evidence of sensitivity bias under the assumption that answering “no” is the sensitive response.

Second, we examine the extent of uniform error and top-biased error in the list experiment. These are examples of “nonstrategic” measurement error; misreporting due to respondent inattentiveness or satisficing (Ahlquist 2018; Blair et al. 2019). Here, we find that respondents may choose random responses in the list experiment as a way of concealing whether or not they have participated in vote buying. This suggests that evidence of uniform error can be interpreted as strategic misreporting. We also find that top-coding – which implies revealing vote buying in the list experiment– is very prevalent, and note that this can be viewed as strategic under the assumption that answering “no” is sensitive. Further, when modeling the correlation between non-sensitive items and the latent sensitive trait, we find that a large proportion of top-coders are likely to truly have participated in vote buying. Taken together, the results of our tests of design suggest that the over-reporting we observe is strategic in nature. This is consistent with theories of vote buying which emphasise its positive connotations, such socially desirability and network benefits. We conclude that ‘design failure’ is not the only reasonable interpretation of unexpected list experiment results.

While there is a substantial amount of prior research employing list experiments to measure vote buying, this paper differs in several specific ways. First, we connect the literature on measurement and the literature on clientelism in order to theorise about the likely reasons for over-reporting vote buying. While the measurement literature identifies diverse sources of sensitivity bias, which manifest in different polarities (Blair et al. 2020; Krumpal 2013), prior research on vote buying has focused almost exclusively on motivations for under-reporting. Second, we apply recent methodological advancements in list experiment diagnostic tests to our list experiment under both assumptions of polarity. Existing publications, in contrast, only present findings under the assumption that vote buying will be under-reported.

The findings have implications for our understandings of clientelism and electoral democracy. While vote-buying has perverse effects on democracy by distorting accountability and violating the fairness of democratic procedures (Stokes 2005), our findings suggest that, in Nigeria, vote buying might in fact be considered legitimate and socially desirable by a significant segment of the population. If over-reporting testifies to norms that not only tolerate, but even encourage vote buying at the level of the individual voter, it might explain the pervasiveness of vote buying practices long after democratic transitions.

The paper also has broader implications. For many sensitive topics, scholars have strong uni-directional assumptions of polarity (Gelman 2014). For example, we expect under-reporting for drug-use and over-reporting for voter turnout. We show how these kinds of assumptions guide what is interpreted as sensitivity bias and what is interpreted as design failure. The implication is that, in some situations, incorrect assumptions could lead to the interpretation of unexpected results as design failures, and to the dismissal of potentially valuable empirical findings. This is relevant for any field of research which relies on self-reporting for the measurement sensitive attitudes and behaviors.

Sensitivity bias is a well-known problem in survey research. The problem is that when survey respondents misreport, survey questions provide inaccurate estimates of aggregate attitudes and behaviors. Because the error in reporting is not random (certain individuals tend to misreport), this creates systematic bias in analyses of the causes and consequences of sensitive topics.

Sensitivity bias is a significant challenge in the social sciences, which often aims to study controversial, taboo, private or otherwise sensitive topics. In the case of vote buying, existing studies provides several reasons why survey respondents may want to avoid reporting participation in vote buying. To begin with, vote buying is illegal in many countries, so respondents may fear legal sanctions (Gonzalez-Ocantos et al. 2012; Mares et al. 2018; Blair et al. 2020; Mares and Young 2019; Stokes 2007). Vote buying may also be socially stigmatised because it violates norms of democratic conduct (Kramon 2016; Corstange 2018), and because of its association with low socio-economic standing (Cammett 2014; Corstange 2018). The existence of social norms which make vote buying socially undesirable also make self-preservation a potential motivation for concealing vote buying: respondents may seek to preserve a positive self-image by denying to themselves that they have violated what they consider to be prevailing norms.

In survey research, these kinds of arguments have motivated the use of list experiments, a technique designed to provide accurate measures of sensitive topics (Blair et al. 2020). List experiments encourage truthful reports of sensitive behaviors by offering respondents greater privacy. Studies using list experiments often also include direct measures of the sensitive topic: standard survey questions without additional privacy. By comparing list experiment estimates to estimates from direct survey questions, scholars are able to detect whether (in the aggregate) respondents answer differently when afforded additional privacy, compared to when asked directly. The difference in estimated prevalence rates is then generally interpreted as evidence of sensitivity bias (Blair and Imai 2012).

In the study of vote buying, sensitivity bias appears common, large in magnitude and consistent in direction. In a meta-analysis of 19 survey studies Blair et al. (2020) conclude that vote buying tends to be under-reported by an average of 8 percentage points. Based both on theory and accumulated evidence, many scholars now assume that vote buying will be under-reported in direct measures (e.g., Blair et al. (2020), Carkoglu and Aytaç (2015, p. 548), Kiewiet De Jonge (2015, p. 720)). However, this prevailing view is partially maintained because studies which do not find evidence of under-reporting tend to be dismissed. It is actually not uncommon that list experiments on vote buying fail to reveal under-reporting (c.f. Castro Cornejo and Beltrán (2020, p. 224)). Returning to Blair et al. (2020)’s meta analysis: 7 out of 19 studies found significant under-reporting, but the remaining 12 studies found no significant discrepancies across a direct measure and a list experiment. Further, 4 of these 12 studies revealed marginally higher prevalence levels in a direct measure, although this over-reporting is not statistically significant. However, as list experiment practitioners have noted, there is a tendency for scholars with null or unexpected list experiment results not to pursue publication, ultimately leading to a file-drawer problem which may slant evidence from meta-analyses (Blair et al. (2020, p. 1306), Kramon and Weghorst (2019, p. 239), Castro Cornejo and Beltrán (2020, p. 224), (Castro Cornejo and Beltrán 2022, p. 10), Gelman (2014)). In the context of vote buying in particular, the assumption that vote buying should be under-reported may cause a kind of confirmation bias where evidence suggesting no sensitivity bias or over-reporting is dismissed and evidence suggesting under-reporting is over-emphasized.⁵ Even those studies which are transparent about discovering significant evidence of over-reporting of vote buying tend to attribute this to respondent error (Castro Cornejo and Beltrán 2020, 2022), design failure or “list experiment breakdown” (Kramon and Weghorst 2019; Kerr 2018).

We consider the tendency to interpret over-reporting as evidence of design failure to be problematic, since what is deemed as a failure depends on our assumptions about the polarity of the sensitive topic: what the sensitive response to the question is, and accordingly, whether under- or over-reporting is the expected indication of sensitivity bias. In the case of vote buying, scholars assume that answering affirmatively when asked about vote buying is the sensitive response; so vote buying will be under-reported in direct survey questions. Uni-directional assumptions are not limited to vote buying For example, Blair et al. (2020) review four political science literatures, identifying for each topic whether under- or over-reporting is expected (see also Gelman (2014)). These uni-directional assumptions seem overly restrictive, considering that the measurement literature identifies a number of different sources of sensitivity bias, which could lead to different polarities for a given topic (Blair et al. 2020; Krumpal 2013).

Based on research on clientelism, we see reasons to reconsider our strong assumptions of polarity when measuring vote buying. Much ethnographic and comparative research suggests that vote buying might not always carry negative social stigma, but also convey benefits, even beyond what is entailed in the specific transaction. To begin with, a vote buying exchange might be embedded in social norms—such as reciprocity and solidarity—which make vote buying a socially desirable behavior. Whereas vote buying within survey research is measured as an isolated exchange close to or on election day, the networks invoked in this form of electoral mobilization oftentimes reflect long-standing relationships between patrons and local communities through which selective benefits such as gifts, services, favors and protection are exchanged for political loyalty (Lawson and Greene 2014). Such patron-client relationships often build on norms of reciprocity in a “moral economy of exchange” (Scott 1972). Research suggests that actors involved in vote-buying transactions (also referred to as election-related gift-giving or favour-rendering) often see them as mutually beneficial, and part of an everyday “problem-solving-network" (e.g., Gonzalez Ocantos et al. 2014; Kiewiet De Jonge 2015; Schaffer 2007). From the perspective of the political candidates, wealth and willingness to give gifts can be associated with authority and legitimacy, and the disbursement of electoral handouts as a sign of civic virtue (Chabal and Daloz 1999; Nugent 2007; Gadjanova 2017). These features also serve to normalize the distribution of electoral handouts amongst voters, and reduces the social stigma associated with it (see e.g., Gadjanova 2017; Nugent 2007; van de Walle 2007). Feelings of obligation, gratitude and norms of reciprocity have been used to explain voters’ compliance with vote-buying offers - even in the presence of ballot secrecy and absent strong monitoring mechanisms (Lawson and Greene 2014; Finan and Schechter 2012). This discussion confirms social desirability and self-presentation as relevant sources of sensitivity bias, but in this case implying that not having participated in vote-buying is the sensitive response, and that vote buying will be over-reported when respondents are asked directly.

To summarise our discussion, Table 1 lists the possible sources of sensitivity bias we have discussed. For each motivation, we list the corresponding polarity assumptions: what the sensitive response to a survey question is and the expected form of sensitivity bias. Blair et al. (2020, p.1300) provides a similar table for vote buying and three other sensitive topics. For vote buying, the motivations Blair et al. (2020) list all result in respondents preferring to answer “no”, with the implication that vote buying will be under-reported. We have additionally theorised about possible motivations which could lead respondents to prefer to answer affirmatively, such as those Blair et al. (2020) and Krumpal (2013) identify for other sensitive topics.

For our empirical analysis, we draw on an original, nationally representative survey of 2400 Nigerian citizens fielded just after the 2019 general elections.⁶ Nigeria represents a suitable case to explore assumptions about the sensitivity of vote buying. On one hand, vote buying is illegal in Nigeria and recent laws trying to curtail its prevalence has highlighted its unlawful and undemocratic aspects (Obe 2019). On the other hand, there is also research suggesting that vote buying and vote selling has become widely accepted norm in Nigeria (Sakariyau et al. 2015) and is deeply intertwined with the nature of electoral politics in the country (Olaniyan 2020). Obe (2019, p. 13), for example remarks on the peculiar honesty of those engaged in vote-selling in Nigeria, noting that many voters in the 2019 elections expected money to show up and openly asked candidates what they were willing to pay. Systematic data from earlier electoral rounds in Nigeria testify to ambiguous popular attitudes to vote buying. Bratton (2008) reports that amongst respondents surveyed by the Afrobarometer after the 2007 election, barely half (49%) thought it was “wrong and punishable” for a Nigerian voter to “accept money in return for a vote”, whereas the other half was willing to excuse participation in a vote buying transaction as “wrong but understandable" (35%) or “not wrong at all" (10%). Comparative studies show that about 28% of Nigerians admit to vote buying in direct survey questions, well above the median rate of 15% (Carkoglu and Aytaç 2015, p. 550), which is another indication that vote buying may be less sensitive or have positive connotations in Nigeria relative to other countries. Our survey asked about political attitudes and election day experiences, and included both a list experiment measure and a direct measure of vote buying.⁷

In the list experiment, respondents were read a list of statements about things they could have experienced during the 2019 election campaign, and were asked to state how many of those things they had experienced, but not which ones. The control group was read a list of four non-sensitive control items (List A in Table 2), while the treatment group was read the same list with the addition of a sensitive item on vote buying (List B in Table 2). By virtue of random assignment, the list experiment measure of vote buying is the difference in the mean count of items across treatment and control groups. The list experiment difference-in-means is an aggregate measure; whether respondents reported vote buying in the list experiment is not observable at the individual level in order to protect respondents’ privacy.

To avoid potential priming effects, we placed the direct measure of vote buying later in the survey after a series of unrelated questions.⁸ In the direct measure, respondents were read a list of statements about ways political actors may have tried to persuade them to vote in a particular way during the 2019 elections, and were asked to state which ones they experienced. The direct measure of vote buying is the proportion of respondents who answered affirmatively to the vote buying item.

When designing the list experiment and direct measure, we strove to ensure that the two questions measure the same quantity, and that this quantity corresponds well to conceptualisations of vote buying. First, both questions ask about an offer of material incentives “to vote in a particular way”. Conceptually, it is important to include a phrase relating to expectations of an exchange in order to capture the quid pro quo nature of vote buying offers (Stokes 2005) and in order to ensure that respondents are reporting a clientelistic exchange (rather than a non-clientelistic exchange, such as the receipt of campaign gifts) (Castro Cornejo and Beltrán 2020, 2022; Kiewiet De Jonge 2015). Second, both questions ask about the receipt of an offer rather than whether respondents accepted the offer. This helps us to capture the socially desirable aspects of vote buying as a clientelistic exchange; the receipt of an offer is likely to be socially desirable in a clientelistic context, while not receiving an offer could be an undesirable signal of exclusion from patron-client networks. Third, both questions ask about respondents’ personal experiences. This is important, as prior research has shown that individuals report vote buying differently when asked about themselves or others (Gonzalez-Ocantos et al. 2012). Fourth, both questions specify an offer of money as prior research has shown that respondents answer differently when asked about money as opposed to other kinds of non-monetary compensation (Kiewiet De Jonge 2015; Castro Cornejo and Beltrán 2020). Our list experiment and direct measures of vote buying are similar to those in Kramon (2016), Gonzalez-Ocantos et al. (2012) and Kiewiet De Jonge (2015).

In addition to our measures of vote buying, we used survey questions on demographic characteristics to construct control variables for our analysis. Drawing on prior related research, we control for age, gender, college education and employment (e.g., Gonzalez-Ocantos et al. 2012; Carkoglu and Aytaç 2015; Castro Cornejo and Beltrán 2020). We additionally include an indicator of whether the respondent is a registered voter, since registered- and non-registered voters received different survey questions.⁹

Following the standard approach to detect sensitivity bias (e.g., Blair et al. 2020; Aronow et al. 2015; Blair and Imai 2012), we compare the proportion of respondents who reported vote buying across the list experiment and the direct measure in our survey. The first plot in Fig. 1 shows reported vote buying in each of the two measures. We plot mean levels of the direct measure over the distribution of responses, to remind us that individual responses are observable in the direct measure, while in the list experiment vote buying can only be calculated in the aggregate. The bottom plot in Fig. 1 shows the estimated difference in reported vote buying comparing the direct measure to the list experiment.

As Fig. 1 shows, we find that survey respondents report higher levels of vote buying in the direct measure compared to the list experiment. This difference is both substantively large, 22 percentage points, and significant at the 95% confidence level.¹⁰

Our exploratory analysis aimed to establish whether the over-reporting of vote buying we observed in our post-election survey can be attributed to sensitivity bias, or whether it is simply the result of a failed list experiment. By submitting the survey data to a series of test of design, we found some evidence that over-reporting is strategic in nature. First, when testing for violations of the “no design effects” and “no liars” assumptions of list experiments, we found no evidence indicative of design failure. Tests for “floor” and “ceiling” effects suggest that respondents do not conceal vote buying in the list experiment. On the contrary, respondents appear to intentionally reveal vote buying. Second, when examining the extent of uniform and top-biased error, we find evidence that respondents strategically select random responses, and that it is likely that who reveal vote buying by top-coding have in fact participated in vote buying. Our conclusion is that the over-reporting we observe is consistent with sensitivity bias in a context where vote buying is considered socially desirable, legitimate or beneficial.

The contribution we make is simultaneously methodological and theoretical: we have shown how our theoretical assumptions about what is desirable and what is stigmatised guide our interpretation of list experiment results. While prior research has shown how standard diagnostic tests can fail to accurately distinguish between strategic and nonstrategic measurement error in indirect questioning techniques (e.g., Kuhn and Vivyan 2021; Höglinger and Jann 2018), our focus is specifically on the role of theoretical assumptions in the interpretation of these tests. In particular, we show how uni-directional assumptions about polarity guide what is considered evidence of strategic or non-strategic error, and consequently, what is interpreted as sensitivity bias or design failure.

These assumptions are particularly strong because they imply uniformity across respondent subgroups (e.g., men and women have the same opinion on what the sensitive response is) and across respondents’ social referents (e.g., individual choices to conceal or exaggerate do not depend on who is asking). We probed some of these assumptions via subgroup analysis, and found initial evidence of non-uniformity. However, we stopped short of confirmatory analyses of subgroup differences in sensitivity bias, given that our list experiment was not designed nor sufficiently powered to detect these differences. We view this an important direction for future research. Considering a range of controversial political topics– vote choice, prejudice, abortion and same-sex marriage– it is challenging to imagine a setting in which different groups agree on what the socially correct response is. Given that uni-directional assumptions about polarity may be problematic, we encourage future research to probe additional assumptions both in relation to theory, and in relation to the analysis of list experiments.