Top

Quality & Quantity

Published in:

11-05-2019

Comparing public communication in democracies and autocracies: automated text analyses of speeches by heads of government

Authors: Seraphine F. Maerz, Carsten Q. Schneider

Published in: Quality & Quantity | Issue 2/2020

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Renewed efforts at empirically distinguishing between different forms of political regimes leave out the cultural dimension. In this article, we demonstrate how modern computational tools can be used to fill this gap. We employ web-scraping techniques to generate a data set of speeches by heads of government in European democracies and autocratic regimes around the globe. Our data set includes 4740 speeches delivered between 1999 and 2019 by 40 political leaders of 27 countries. By scaling the results of a dictionary application, we show how, in comparative terms, liberal or illiberal the leaders present themselves to their national and international audience. In order to gauge whether our liberalness scale reveals meaningful distinctions, we perform a series of validity tests: criterion validity, qualitative hand-coding, unsupervised topic modeling, and network analysis. All tests suggest that our liberalness scale does capture meaningful differences between political regimes despite the large heterogeneity of our data.

previous article Co-developing beliefs and social influence networks—towards understanding socio-cognitive processes like Brexit

next article The moral–emotional foundations of political discourse: a comparative analysis of the speech records of the U.S. and the Japanese legislatures

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Boese (2019) provides a thorough comparison of V-Dem with Freedom House and Polity and highlights these and other advantages of the most recent V-Dem data set.

Likewise, if public communication by an autocrat continuously emphasizes liberal political norms and values, then, we argue, it undermines the persistence of the illiberal autocratic regime.

With this, we, obviously, do not claim that political leadership (and its rhetoric) is identical to the political regime (and its institutional practices). Instead, we follow those in the literature who argue that the former is prone to have an impact on the longer-term viability of the latter (Diamond 1999; Linz and Stepan 1996; Merkel 1998; Higley and Burton 2006).

We refer to a rather broad understanding of political speeches here which occasionally includes also political statements, e.g. from press conferences, and other more spontaneously produced documents.

To formally distinguish between both regime types, we rely on the most recent V-Dem data and their regime typology (Coppedge et al. 2018; Lührmann et al. 2018b, see also Sect. 4.1).

de Vries et al. (2018) illustrate in various tests that for bag-of-words text models findings generated from human-translated and machine-translated texts highly overlap. Lucas et al. (2015) provide a showcase of how to preprocess and manage multilingual texts in R.

This is demonstrated by the fact that among liberal democracies economic models with varying economic liberties can be found.

See “Appendix” section for the complete list of our dictionary terms.

Another constraint of the model is that it does not account for differences in rhetoric over time. Yet, this is rather a problem of data availability since we do not have enough speeches in each case for estimates per year.

The replication files, including robustness tests for different pre-processing strategies, are available at: https://dataverse.harvard.edu/dataverse/sfm.

For this formal distinction between democracy and autocracy we rely on the most recent data of the V-Dem Project (Coppedge et al. 2018; Lührmann et al. 2018a) who classify autocracies as regimes in which no de-facto multiparty, or free and fair elections, or Dahl’s institutional prerequisites are not minimally fulfilled (Lührmann et al. 2018b).

Concerning the current Polish prime minister Mateusz Morawiecki and Molodiva’s Pavel Filip, we cannot make clear classifications because the confidence intervals of their point estimates cross the zero line (cf. Fig. 1). The same is true for Edi Rama from Albania—yet, in his case the confidence interval overlaps only by a very small margin, suggesting that he is rather to be seen as an illiberal than liberal speaker.

Available here: http://www.kormany.hu/en/the-prime-minister/the-prime-minister-s-speeches/prime-minister-viktor-orban-s-speech-at-the-final-fidesz-election-campaign-event.

Available here: https://www.bundeskanzlerin.de/bkin-en/news/towards-a-cosmopolitan-and-diverse-germany-1141998.

For an exploratory study on different language styles among autocrats, see Maerz (2019).

We heavily pre-processed our corpus before applying the unsupervised techniques (removal of stop words, infrequently used terms, punctuation, numbers, stemming, lowercase) to remove irrelevant words and treat words with similar properties as identical. As recommended by Denny and Spirling (2018) and illustrated in our “Appendix” section, we conducted detailed robustness tests to make sure that none of these preprocessing steps uncontrollably alters the STM results. All operations were done in R (2019, v. 3.5.2.) with the STM package (Roberts et al. 2015, v. 1.3.3.). The replication files are available here: https://dataverse.harvard.edu/dataverse/sfm.

Vladimir Putin has the largest share and smallest 95% confidence interval in this topic, cf. Fig. 4 in the “Appendix” section.

Since the model measures relative topic proportions, the rather isolated position of Orbán in this regard is not a consequence of his comparatively large share of speeches in the corpus. Other speakers have similarly isolated positions despite their rather small amount of speeches (e.g. Emmanuel Macron on ‘Collective Memory’ in Fig. 5, “Appendix” section, or Kim Jong Un on ‘Juche, Military’ in Fig. 4).

The order of the speakers in these plots is based on their scores on our dictionary scale, the horizontal lines around the point estimates refer to the 95% confidence interval for the relative proportions of each speaker.

The way Orbán’s government attacks George Soros, a financier and philanthropist known for his pro-migration and liberal opinions, is a case sui generis in the European Union. Orban’s most recent election campaign has been broadly understood as an anti-Jewish and anti-Muslim manifesto, for example by Cohen (2018) in the Guardian.

For the specifics of illiberal and autocratic language styles, see Dowell et al. (2015), Windsor et al. (2015, 2017) and Maerz (2019).

Lucas et al. (2015, Appendix E) provide more details about the graph estimation procedure which we have adopted here.

Details can be found in the material available at https://dataverse.harvard.edu/dataverse/sfm.

Further information also here: http://www.mjdenny.com/getting_started_with_preText.html.

Adcock, R., Collier, D.: Measurement validity: a shared standard for qualitative and quantitative research. Am. Polit. Sci. Rev. 95(3), 529–546 (2001)CrossRef

Arat, Z.F.: Democracy and Human Rights in Developing Countries. Lynne Rienner, Boulder (1991)

Benoit, K., Nulty, P., Obeng, A., Wang, H., Lauderdale, B., Lowe, W.: Quanteda package (2019). Retrieved from https://cran.r-project.org/web/packages/quanteda/quanteda.pdf

Bocskor, Á.: Anti-immigration discourses in Hungary during the ‘Crisis’ year: The Orbân Government’s ‘National Consultation’ Campaign of 2015. Sociology 52(3), 551–568 (2018)CrossRef

Boese, V.A.: How (not) to measure democracy. Int. Area Stud. Rev. (2019)

Bogaards, M.: De-democratization in Hungary: diffusely defective democracy. Democratization (2018). https://doi.org/10.1080/13510347.2018.1485015 CrossRef

Bollen, K.A.: Issues in the comparative measurement of political democracy. Am. Sociol. Rev. 45(3), 370–390 (1980)CrossRef

Bozóki, A., Hegedűs, D.: An externally constrained hybrid regime: Hungary in the European Union. Democratization (2018). https://doi.org/10.1080/13510347.2018.1455664 CrossRef

Chang, J., Gerrish, S., Wang, C., Blei, D.M.: Reading tea leaves: how humans interpret topic models. Adv. Neural Inf. Process. Syst. 22, 288–296 (2009)

Cianetti, L., Dawson, J., Hanley, S.: Rethinking “democratic backsliding” in Central and Eastern Europe—looking beyond Hungary and Poland. East Eur. Polit. 34(3), 243–256 (2018). https://doi.org/10.1080/21599165.2018.1491401 CrossRef

Cohen, N.: In Hungary, the exploitation of a mythical enemy is poisoning politics. The Guardian, 31 Mar. 2018. (2018) Retrieved from https://www.theguardian.com/commentisfree/2018/mar/31/in-hungary-exploitation-of-mythical-enemy-is-poisoning-politics

Coppedge, et al.: V-dem country-year dataset v8, Varieties of Democracy (V-Dem) Project (2018). https://doi.org/10.23696/vdemcy18

Dahl, R.A.: Polyarchy: participation and opposition. Yale University Press, New Haven (1971)

Dahl, R.A.: Democracy and its critics. Yale University Press, New Haven (1989)

Denny, M.J., Spirling, A.: Text preprocessing for unsupervised learning: why it matters, when it misleads, and what to do about it. Polit. Anal. 26, 168–189 (2018)CrossRef

de Vries, E., Schoonvelde, M., Schumacher, G.: No longer lost in translation. Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications, Political Analysis, Online First (2018). Retrieved from https://doi.org/10.1017/pan.2018.26

Diamond, L.J.: Developing Democracy—Toward Consolidation. The John Hopkins University Press, Baltimore (1999)

Dowell, N.M., Windsor, L.C., Graesser, A.C.: Computational linguistics analysis of leaders during crises in authoritarian regimes. Dyn. Asymmetric Confl. 9(01–03), 1–12 (2015). https://doi.org/10.1080/17467586.2015.1038286 CrossRef

Dukalskis, A.: The Authoritarian Public Sphere—Legitimation and Autocratic Power in North Korea, Burma, and China. Routledge, London (2017)

Dukalskis, A., Gerschewski, J.: What autocracies say (and what citizens hear): proposing four mechanisms of autocratic legitimation. Contemp. Polit. 23, 251–268 (2017)CrossRef

Easton, D.: A Systems Analysis of Political Life. Wiley, New York (1965)

Fauve, A.: Global Astana: nation branding as a legitimization tool for authoritarian regimes. Cent. Asian Surv. 34, 110–124 (2015). https://doi.org/10.1080/02634937.2015.1016799 CrossRef

FreedomHouse: Freedom house on Hungary in 2019 (2019). Retrieved from https://freedomhouse.org/report/freedom-world/2019/hungary

Geddes, B., Frantz, E., Wright, J.: Autocratic breakdown and regime transitions: a new data set. Perspect. Polit. 12(02), 313–331 (2014)CrossRef

Gerschewski, J.: The three pillars of stability: legitimation, repression, and cooptation in autocratic regimes. Democratization 20(1), 13–38 (2013)CrossRef

Greene, D., O’Callaghan, D., Cunningham, P.: How many topics? Stability analysis for topic models. Lecture Notes in Computer Science, 8724 LNAI (PART 1), pp. 498–513 (2014)

Grimmer, J., Stewart, B.M.: Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Polit. Anal. 21(03), 267–297 (2013)CrossRef

Hanley, S., Vachudova, M.A.: Understanding the illiberal turn: democratic backsliding in the Czech Republic*. East Eur. Polit. 34(3), 276–296 (2018). https://doi.org/10.1080/21599165.2018.1493457 CrossRef

Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1(1), 77–89 (2007). https://doi.org/10.1080/19312450709336664 CrossRef

Higley, J., Burton, M.: Elite Foundations of Liberal Democracy. Rowman and Littlefield Publishers, Oxford (2006)

Inglehart, R., Welzel, C.: Modernization, Cultural Change, and Democracy: The Human Development Sequence. Cambridge University Press, Cambridge (2005)

Kirsch, H., Welzel, C.: Democracy misunderstood: authoritarian notions of democracy around the globe. Soc. Forces (2018). https://doi.org/10.1093/sf/soy114 CrossRef

Krekó, P., Enyedi, Z.: Orbán ’ s laboratory of illiberalism. J. Democr. 29(3), 39–51 (2018)CrossRef

Langer, A.I., Sagarzazu, I.: Are all policy decisions equal? Explaining the variation in media coverage of the UK budget. Policy Stud. J. 45(2), 337–358 (2015). https://doi.org/10.1111/psj.12119 CrossRef

Laver, M., Benoit, K., Garry, J.: Extracting policy positions from political texts using words as data. Am. Polit. Sci. Rev. 97(2), 311–331 (2003)CrossRef

Linz, J.J., Stepan, A.C.: Problems of Democratic Transition and Consolidation: Southern Europe, South America, and Post-communist Europe. Johns Hopkins University Press, Baltimore (1996)

Lowe, W.: Yoshikoder: cross-platform multilingual content analysis. Java Software Version 0.6.5. (2015). Retrieved from http://yoshikoder.sourceforge.net/index.html

Lowe, W., Benoit, K., Mikhaylov, S., Laver, M.: Scaling policy preferences from coded political texts. Legis. Stud. Q. 36, 123–155 (2011)CrossRef

Lucas, C., Nielsen, R.A., Roberts, M.E., Stewart, B.M., Storer, A., Tingley, D.: Computer-assisted text analysis for comparative politics. Polit. Anal. 23, 254–277 (2015)CrossRef

Lührmann, A., Mechkova, V., Dahlum, S., Maxwell, L., Petrarca, C.S., Sigman, R., Staffan, I.: State of the world 2017: Autocratization and exclusion? Democratization (2018a). https://doi.org/10.1080/13510347.2018.1479693 CrossRef

Lührmann, A., Tannenberg, M., Lindberg, S.I.: Regimes of the world (RoW): opening new avenues for the comparative study of political regimes. Polit. Gov. 6(1), 60 (2018b)

Maerz, S.F.: Ma’naviyat in Uzbekistan: An ideological extrication from its soviet Past? J. Polit. Ideol. 23, 205–222 (2018a). https://doi.org/10.1080/13569317.2018.1419448 CrossRef

Maerz, S.F.: The many faces of authoritarian persistence. A set-theory perspective on the survival strategies of authoritarian regimes. Gov. Oppos. (2018b). https://doi.org/10.7910/DVN/DZ7JLC CrossRef

Maerz, S.F.: Simulating pluralism: the language of democracy in hegemonic authoritarianism. Polit. Res. Exch. (2019). https://doi.org/10.1080/2474736X.2019.1605834 CrossRef

Maerz, S.F., Puschmann, C.: Text as data and automated content analysis for conflict research: a literature survey. In: Computational Conflict Research (Springer Nature as Part of Their Computational Social Sciences Series) (forthcoming) (2019)

Makarychev, A., Yatsyk, A.: Entertain and govern: from sochi 2014 to FIFA 2018. Probl. Post Communism 65(2), 115–128 (2018)CrossRef

Megoran, N.: Framing Andijon, narrating the nation: Islam Karimov’s account of the events of 13 May 2005. Cent. Asian Surv. 27(1), 15–31 (2008). https://doi.org/10.1080/02634930802213965 CrossRef

Merkel, W.: The consolidation of post-autocratic democracies: a multi-level model. Democratization 5(3), 33–67 (1998). https://doi.org/10.1080/13510349808403572 CrossRef

Munzert, S., Rubba, C., Meissner, P., Nyhuis, D.: Automated Data Collection with R: A Practical Guide to Web Scraping and Text Mining. Wiley, London (2015)

Nelson, L.K., Burk, D., Knudsen, M., McCall, L.: The future of coding: a comparison of hand-coding and three types of computer-assisted text analysis methods. Sociol. Methods Res. (2018). https://doi.org/10.1177/0049124118769114 CrossRef

Omelicheva, M.Y.: Authoritarian legitimation: assessing discourses of legitimacy in Kazakhstan and Uzbekistan. Cent. Asian Surv. 35(4), 481–500 (2016). https://doi.org/10.1080/02634937.2016.1245181 CrossRef

R Core Team: R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna (2019). Retrieved from http://www.r-project.org

Roberts, M.E., Stewart, B.M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S.K., Rand, D.G.: Structural topic models for open-ended survey responses. Am. J. Polit. Sci. 58(4), 1064–1082 (2014)CrossRef

Roberts, M.E., Stewart, B.M., Tingley, D., Benoit, K.: Package ‘stm’ (2015). Retrieved from https://cran.r-project.org/web/packages/stm/stm.pdf

Roberts, M.E., Stewart, B.M., Tingley, D.: stm: R package for structural topic models. J. Stat. Softw. 1, 12 (2018)

Scheppele, K.L.: The rule of law and the Frankenstate: why governance checklists do not work. Governance 26(4), 559–562 (2013). https://doi.org/10.1111/gove.12049 CrossRef

Schumpeter, J.A.: Capitalism, Socialism and Democracy, 1st edn. Harper Brothers, New York (1942)

Stanley, B.: Confrontation by default and confrontation by design: strategic and institutional responses to Poland’s populist coalition government. Democratization 23(2), 263–282 (2016)CrossRef

Vanhanen, T.: Global trends of democratization in the 1990s: a statistical analysis, Berlin (1994)

Welzel, C.: Freedom Rising: Human Empowerment and the Quest for Emancipation. Cambridge University Press, Cambridge (2013)CrossRef

Windsor, L., Dowell, N., Graesser, A.: The language of autocrats: leaders’ language in natural disaster crises. Risk Hazards Crisis Public Policy 5(4), 446–467 (2015)CrossRef

Windsor, L., Dowell, N., Windsor, A., Kaltner, J.: Leader language and political survival strategies. Int. Interact. 44(2), 321–336 (2017). https://doi.org/10.1080/03050629.2017.1345737 CrossRef

Zakaria, F.: The rise of illiberal democracy. Foreign Aff. 76(6), 22–43 (1997)CrossRef

Title: Comparing public communication in democracies and autocracies: automated text analyses of speeches by heads of government
Authors: Seraphine F. Maerz
Carsten Q. Schneider
Publication date: 11-05-2019
Publisher: Springer Netherlands
Published in: Quality & Quantity / Issue 2/2020
Print ISSN: 0033-5177
Electronic ISSN: 1573-7845
DOI: https://doi.org/10.1007/s11135-019-00885-7

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 2/2020

Comparing supervised learning algorithms and artificial neural networks for conflict prediction: performance and applicability of deep learning in the field

Co-developing beliefs and social influence networks—towards understanding socio-cognitive processes like Brexit

Political culture research: dilemmas and trends. Prologue to the special issue

A network agent-based model of ethnocentrism and intergroup cooperation

Correction to: The stories groups tell: campaign finance reform and the narrative networks of cultural cognition

Enclave deliberation and common-pool resources: an attempt to apply Civic Preference Forum on community gardening in Hungary