Top

Political Behavior

Published in:

28-03-2016 | Original Paper

Unresponsive and Unpersuaded: The Unintended Consequences of a Voter Persuasion Effort

Authors: Michael A. Bailey, Daniel J. Hopkins, Todd Rogers

Published in: Political Behavior | Issue 3/2016

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

To date, field experiments on campaign tactics have focused overwhelmingly on mobilization and voter turnout, with far more limited attention to persuasion and vote choice. In this paper, we analyze a field experiment with 56,000 Wisconsin voters designed to measure the persuasive effects of canvassing, phone calls, and mailings during the 2008 presidential election. Focusing on the canvassing treatment, we find that persuasive appeals had two unintended consequences. First, they reduced responsiveness to a follow-up survey among infrequent voters, a substantively meaningful behavioral response that has the potential to induce bias in estimates of persuasion effects as well. Second, the persuasive appeals possibly reduced candidate support and almost certainly did not increase it. This counterintuitive finding is reinforced by multiple statistical methods and suggests that contact by a political campaign may engender a backlash.

previous article National Party Division and Divisive State Primaries in U.S. Presidential Elections, 1948–2012

next article Negative Advertising and the Dynamics of Candidate Support

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

The data set and replication code are posted online at https://dataverse.harvard.edu/dataverse/DJHopkins. Due to their proprietary nature, two variables employed in our analyses are omitted from the data set: the Democratic performance in a precinct and each respondent’s probability of voting for the Democratic candidate.

Strategies to study persuasion include natural experiments based on the uneven mapping of television markets to swing states (Simon and Stern 1955; Huber and Arceneaux 2007) or the timing of campaign events (Ladd and Lenz 2009). Other studies use precinct-level randomization (e.g. Arceneaux 2005; Panagopoulos and Green 2008; Rogers and Middleton 2015) or discontinuities in campaigns’ targeting formulae (e.g. Gerber et al. 2011).

In a related vein, Shi (2015) finds that postcards exposing voters to a dissonant argument on same-sex marriage reduce subsequent voter turnout.

Experimental studies also rely on self-reported vote choice, not the actual vote cast. This is less of a concern, as pre-election public opinion surveys like this one typically provide accurate measures of vote choice (Hopkins 2009).

Such support scores are commonly employed by campaigns. To generate them, data vendors fit a model to data where candidate support is observed, typically survey data. They then use the model, alongside known demographic and geographic characteristics, to estimate each voter’s probability of supporting a given candidate in a much broader sample. The specific model employed is proprietary and unknown to the researchers. The Pearson’s correlation with a separate measure of precinct-level prior Democratic support is 0.47, indicating the importance of precinct-level measures in its calculation in this data set. For more on the use of such data and scores within political science, see Ansolabehere et al. (2011), Ansolabehere et al. (2012), Rogers and Aida (2014) and Hersh (2015).

This age skew reduces one empirical concern, which is that voters under the age of 26 have truncated vote histories. Only 2.1% of targeted voters were under 26 in 2008, and thus under 18 in 2000.

Specifically, voters were coded as “strong Obama,” “lean Obama,” “undecided,” “lean McCain,” and “strong McCain.”

We can do additional analyses to approximate the effect of the treatment on people who actually spoke to the canvassers (the so-called Complier Average Causal Effect; see Angrist et al. 1996), and report the results in the Conclusion.

For the full regression, see the first column of Table 6 in the Appendix.

Similar results for the phone and mail treatments show no significant differences across groups.

For the corresponding regression model, see the second column of Table 6 in the Appendix.

Voters under the age of 26 would not have been eligible to vote in some of the prior elections, and might be disproportionately represented among the low-turnout groups. We have age data only for 39, 187 individuals in the sample. The negative effects of canvassing in the zero-turnout group persist (with a larger confidence interval) when the data set is restricted to citizens known to be older than 26.

The effects for phone calls are generally similar, but not statistically significant (see Table 9 in the Appendix).

For example, Enos et al. (2014) find that direct mail, phone calls, and canvassing had small effects on turnout for voters with low probabilities of voting, large effects for voters with middle-to-high probabilities of voting, and smaller but still positive effects for those with the highest probabilities of voting.

Results using logistic regression are highly similar.

In separate, ongoing research, we use the turnout results described above as a benchmark with which to evaluate each of these methods.

As Little et al. (2012) explain, “weighted estimating equations and multiple-imputation models have an advantage in that they can be used to incorporate auxiliary information about the missing data into the final analysis, and they give standard errors and p values that incorporate missing-data uncertainty”(1359).

But that fact also means that the “implied joint distributions may not exist theoretically”(Buuren et al. 2006, p. 1051). Still, that important theoretical limitation does not prevent MICE from working well in practice (Buuren et al. 2006).

To examine the performance of our model for multiple imputation, we performed tests in which we deliberately deleted 500 known survey responses from the fully observed data set (n = 12,442) and then assessed the performance of our imputation model for those 500 cases where we know the correct answer. In each case, we used the full multiple imputation model to generate five imputed data sets for each new data set, and then calculated the share of deleted responses which we correctly imputed. The median out-of-sample accuracy across the resulting data sets was 74.9 %, with a minimum of 73.3 % and a maximum of 76.0 %. This performance is certainly better than chance alone.

In fact, the associated p value is less than 0.002, meaning that the finding would remain significant even after a Bonferroni correction for multiple comparisons to account for the analyses of the phone and mail treatments.

The associated 95 % confidence interval spans from −3.03 to −0.60.

We could add additional covariates that only affect this equation without affecting our discussion below. The existence of such variables is commonly necessary for empirical estimation of selection models, although it is not strictly required, as these models can be identified solely with parametric assumptions about error terms.

Throughout these analyses, we drop our measure of respondents’ age, which is the only independent variable with significant missingness.

Here, \(\delta\) is set to 0.0001.

Siddique and Belin (2008a) report that a value of \(k=3\) works well in their substantive application, while Siddique and Belin (2008b) recommend values between 1 and 2.

Still, even in light of this potential to under-estimate variance, Demirtas et al. (2007) demonstrate that the small-sample properties of the original ABB are superior when compared to would-be corrections.

IPW requires data that are fully observed with the exception of the missing outcome. We thus set aside 20 respondents who were missing data for covariates other than age or Obama support.

Adams, W. C., & Smith, D. J. (1980). Effects of telephone canvassing on turnout and preferences: A field experiment. Public Opinion Quarterly, 44(3), 389–395.CrossRef

Albertson, B., & Busby, J. W. (2015). Hearts or minds? Identifying persuasive messages on climate change. Research & Politics.

Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables (with discussion). Journal of the American Statistical Association, 91, 444–455.CrossRef

Ansolabehere, S., & Hersh, E. (2011). Who really votes? In P. M. Sniderman & B. Highton (Eds.), Facing the challenge of democracy: Explorations in the analysis of public opinion and political participation. Princeton University Press.

Ansolabehere, S., & Hersh, E. (2012). Validation: What big data reveal about survey misreporting and the real electorate. Political Analysis, 20(4):437–459.

Arceneaux, K. (2005). Using cluster randomized field experiments to study voting behavior. The Annals of the American Academy of Political and Social Science, 601(1), 169–179.CrossRef

Arceneaux, K. (2007). I’m asking for your support: The effects of personally delivered campaign messages on voting decisions and opinion formation. Quarterly Journal of Political Science, 2(1), 43–65.CrossRef

Arceneaux, K., & Kolodny, R. (2009). Educating the least informed: Group endorsements in a grassroots campaign. American Journal of Political Science, 53(4), 755–770.CrossRef

Arceneaux, K., & Nickerson, D. W. (2009). Who is mobilized to vote? A re-analysis of 11 field experiments. American Journal of Political Science, 53(1), 1–16.CrossRef

Bechtel, M. M., Hainmueller, J., Hangartner, D., & Helbling, M. (2014). Reality bites: The limits of framing effects for salient and contested policy issues. Political Science Research and Methods (forthcoming).

Broockman, D. E., & Green, D. P. (2014). Do online advertisements increase political candidates’ name recognition or favorability? evidence from randomized field experiments. Political Behavior, 36, 263–289.CrossRef

Cardy, E. A. (2005). An experimental field study of the GOTV and persuasion effects of partisan direct mail and phone calls. The Annals of the American Academy of Political and Social Science, 601(1), 28–40.CrossRef

Cranmer, S. J., & Gill, J. (2013). We have to be discrete about this: A non-parametric imputation technique for missing categorical data. British Journal of Political Science, 43(2), 425–449.CrossRef

Das, M., Newey, W. K., & Vella, F. (2003). Nonparametric estimation of sample selection models. The Review of Economic Studies, 70(1), 33–58.CrossRef

Demirtas, H., Arguelles, L. M., Chung, H., & Hedeker, D. (2007). On the performance of bias-reduction techniques for variance estimation in approximate Bayesian bootstrap imputation. Computational Statistics & Data Analysis, 51(8), 4064–4068.CrossRef

Enos, R. D., Fowler, A., & Vavreck, L. (2014). Increasing inequality: The effect of GOTV mobilization on the composition of the electorate. The Journal of Politics, 76(1), 273–288.CrossRef

Enos, R. D., & Hersh, E. D. (2015). Party activists as campaign advertisers: The ground campaign as a principal-agent problem. American Political Science Review, 109(02), 252–278.CrossRef

Gerber, A., Karlan, D., & Bergan, D. (2009). Does the media matter? A field experiment measuring the effect of newspapers on voting behavior and political opinions. American Economic Journal: Applied Economics, 1(2), 35–52.

Gerber, A., & Green, D. (2000). The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment. American Political Science Review, 94(3), 653–663.CrossRef

Gerber, A. S., Kessler, D. P., & Meredith, M. (2011). The persuasive effects of direct mail: A regression discontinuity based approach. Journal of Politics, 73(1), 140–155.CrossRef

Gerber, A. S., & Green, D. P. (2012). Field experiments: Design, analysis, and interpretation. New York, NY: W.W. Norton and Company.

Gerber, A. S., Huber, G. A., Doherty, D., Dowling, C. M., & Hill, S. J. (2013). Who wants to discuss vote choices with others? Polarization in preferences for deliberation. Public Opinion Quarterly, 77(2), 474–496.CrossRef

Gerber, A. S., Huber, G. A., & Washington, E. (2010). Party affiliation, partisanship, and political beliefs: A field experiment. American Political Science Review, 104(04), 720–744.CrossRef

Gerber, A. S., Gimpel, J. G., Green, D. P., & Shaw, D. R. (2011). How large and long-lasting are the persuasive effects of televised campaign ads? Results from a randomized field experiment. American Political Science Review, 105(01), 135–150.CrossRef

Glynn, A. N., & Quinn, K. M. (2010). An introduction to the augmented inverse propensity weighted estimator. Political Analysis, 18(1), 36–56.CrossRef

Green, D. P., & Gerber, A. S. (2008). Get out the vote: How to increase voter turnout. Washington, DC: Brookings Institution Press.

Heckman, J. (1976). The common structure of statistical models of truncation, sample selectionand limited dependent variables, and simple estimator for such models. Annals of Economic and Social Measurement, 5, 475–492.

Hersh, E. D. (2015). Hacking the electorate: How campaigns perceive voters. New York, NY: Cambridge University Press.CrossRef

Hersh, E. D., & Schaffner, B. F. (2013). Targeted campaign appeals and the value of ambiguity The Journal of Politics, 75(02), 520–534.CrossRef

Hopkins, D. J. (2009). No more wilder effect, never a Whitman effect: When and why polls mislead about black and female candidates. The Journal of Politics, 71(3), 769–781.CrossRef

Huber, G. A., & Arceneaux, K. (2007). Identifying the persuasive effects of presidential advertising. American Journal of Political Science, 51(4), 957–977.CrossRef

Imai, K., King, G., & Stuart, E. A. (2008). Misunderstandings between experimentalists and observation lists about causal inference. Journal of the Royal Statistical Society: Series A, 171(2), 481–502.CrossRef

Issenberg, S. (2012). Obama Does It Better. Slate.

King, G., Honaker, J., Joseph, A., & Scheve, K. (2001). Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review, 95(1), 49–69.

Ladd, J. M., & Lenz, G. S. (2009). Exploiting a rare communication shift to document the persuasive power of the news media. American Journal of Political Science, 53(2), 394–410.CrossRef

Little, R. J., D’Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., et al. (2012). The prevention and treatment of missing data in clinical trials. New England Journal of Medicine, 367(14), 1355–1360.CrossRef

Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). New York, NY: Wiley.

Matland, R. E., & Murray, G. R. (2013). An experimental test for backlash against social pressure techniques used to mobilize voters. American Politics Research, 41(3), 359–386.CrossRef

Michelson, M. R. (2014). Memory and voter mobilization. Polity, 46, 591–610.CrossRef

Moore, R. T. (2012). Multivariate continuous blocking to improve political science experiments. Political Analysis, 20(4), 460–479.CrossRef

Nicholson, S. P. (2012). Polarizing cues. American Journal of Political Science, 56(1), 52–66.CrossRef

Nickerson, D. W. (2005a). Partisan mobilization using volunteer phone banks and door hangers. The Annals of the American Academy of Political and Social Science, 601(1), 10–27.CrossRef

Nickerson, D. W. (2005b). Scalable protocols offer efficient design for field experiements Political Analysis, 13, 233–252.CrossRef

Nickerson, D. W. (2008). Is voting contagious? Evidence from two field experiments. American Political Science Review, 102(1), 49.CrossRef

Nickerson, D. W., & Rogers, T. (2010). Do you have a voting plan? Implementation intentions, voter turnout, and organic plan making. Psychological Science, 21(2), 194–199.CrossRef

Panagopoulos, C., & Green, D. P. (2008). Field experiments testing the impact of radio advertisements on electoral competition. American Journal of Political Science, 52(1), 156–168.CrossRef

Rogers, T., & Nickerson, D. (2013). Can inaccurate beliefs about incumbents be changed? And can reframing change votes? HKS Faculty Research Working Paper Series RWP13-018.

Rogers, T., & Middleton, J. A. (2015). Are ballot initiative outcomes influenced by the campaigns of independent groups? A precinct-randomized field experiment showing that they are. Political Behavior, 37, 567–593.

Rogers, T., & Aida, M. (2014). Vote self-prediction hardly predicts who will vote, and is (misleadingly) unbiased. American Politics Research, 42(3), 503–528.CrossRef

Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied Statistics, 2, 808–840.

Rubin, D. B., & Schenker, N. (1991). Multiple imputation in health-care databases: An overview and some applications. Statistics in Medicine, 10(4), 585–598.CrossRef

Rubin, D., & Schenker, N. (1986). Multiple imputation for interval estimation for simple random samples with ignorable nonresponse. Journal of the American Statistical Association, 81(394), 366–374.CrossRef

Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall.CrossRef

Shi, Y. (2015). Cross-cutting messages and voter turnout: Evidence from a same-sex marriage amendment. Political Communication. (forthcoming).

Siddique, J., & Belin, T. R. (2008a). Multiple imputation using an iterative hot-deck with distance-based donor selection. Statistics in Medicine, 27(1), 83–102.CrossRef

Siddique, J., & Belin, T. R. (2008b). Using an approximate Bayesian bootstrap to multiply impute nonignorable missing data. Computational Statistics & Data Analysis, 53(2), 405–415.CrossRef

Simon, H. A., & Stern, F. (1955). The effect of television upon voting behavior in Iowa in the 1952 presidential election. American Political Science Review, 49(2), 470–477.CrossRef

Sinclair, B. (2012). The social citizen. Chicago, IL: University of Chicago Press.CrossRef

Sinclair, B., McConnell, M., & Green, D. P. (2012). Detecting spillover effects: Design and analysis of multilevel experiments. American Journal of Political Science, 56(4), 1055–1069.CrossRef

Taber, C. S., & Lodge, M. (2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50(3), 755–769.CrossRef

Van Buuren, S., Brand, J. P. L., Groothuis-Oudshoorn, C. G. M., & Rubin, D. B. (2006). Fully conditional specification in multivariate imputation. Journal of Statistical Computation and Simulation, 76(12), 1049–1064.CrossRef

Vavreck, L., et al. (2007). The exaggerated effects of advertising on turnout: The dangers of self-reports Quarterly Journal of Political Science, 2(4), 325–343.CrossRef

Westfall, P. H., & Young, S. S. (1993). Resampling-based multiple testing: Examples and methods for p-value adjustment. New York, NY: Wiley.

Zaller, J. R. (1992). The nature and origins of mass opinion. New York, NY: Cambridge University Press.CrossRef

Title: Unresponsive and Unpersuaded: The Unintended Consequences of a Voter Persuasion Effort
Authors: Michael A. Bailey
Daniel J. Hopkins
Todd Rogers
Publication date: 28-03-2016
Publisher: Springer US
Published in: Political Behavior / Issue 3/2016
Print ISSN: 0190-9320
Electronic ISSN: 1573-6687
DOI: https://doi.org/10.1007/s11109-016-9338-8

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3/2016

Intra-Party Politics and Public Opinion: How Candidate Selection Processes Affect Citizens’ Satisfaction with Democracy

Rolling off the Tongue into the Top-of-the-Head: Explaining Language Effects on Public Opinion

National Party Division and Divisive State Primaries in U.S. Presidential Elections, 1948–2012

Are Group Cues Necessary? How Anger Makes Ethnocentrism Among Whites a Stronger Predictor of Racial and Immigration Policy Opinions

Run, Jane, Run! Gendered Responses to Political Party Recruitment

Dead Man Walking: The Affective Roots of Issue Proximity Between Voters and Parties