Top

Quality & Quantity

Published in:

01-11-2015

An iterative expert survey approach for estimating parties’ policy positions

Author: Kostas Gemenis

Published in: Quality & Quantity | Issue 6/2015

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

This article introduces the iterative expert survey approach in estimating parties’ policy positions. Methodologically, the proposed approach is based on the tradition of ‘judgemental’ coding in the content analysis of political text, and incorporates the idea of anonymous iteration among a panel of expert coders taken from the method known as ‘Delphi’. Anonymous iteration presents an effective way of reducing the random error, and potential bias arising from inter-expert/coder disagreement evident in other popular methods. I provide an empirical demonstration of the approach by estimating parties’ policy positions in the context of a voting advice application in Germany, and argue that the method has considerable potential to generate valid and reliable data on party positions cross-nationally and retrospectively.

previous article An interval fuzzy land-use allocation model (IFLAM) for Beijing in association with environmental and ecological consideration under uncertainty

next article Statistical assessment of regional wealth inequalities: the Italian case

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

The scaling critiques of the Manifesto Project are too many to be referenced here. Most are reviewed in Gemenis (2013a), while Dinas and Gemenis (2010) compare some of the most popular scaling methods in terms of their validity and reliability.

This problem has also been identified in mass surveys where voters’ lack of knowledge leads to implausible centrist estimates of parties that are known to be extreme (Tilley and Wlezien 2008).

A third possibility occurs when the panellist is convinced that the estimate he or she has given in the first round was correct. In this case, the panellist can provide an estimate towards the extremes of the scale in an attempt to influence the group average.

There is disagreement whether panellists should be remunerated. Meyer and Booker (1991, pp. 89–90) generally advise against remuneration. Future studies may use remuneration for graduate students, but not for panellists who are considered to have substantial expertise and could otherwise be motivated in participating because of their interest in the intrinsic aspects of the project.

In their meta-analysis of evaluative studies of the Delphi method, Rowe and Wright (1999) reported that only three out of 27 studies reviewed used panels with fewer than five panellists.

Other studies opted to operationalize confidence in estimation using self-rated measures of expertise. These measures produced inconclusive results in Delphi studies (see Rowe and Wright 1999, pp. 371–372), or have failed to establish a link between self-rated expertise and distance from the group average in conventional expert surveys (Albright and Mair 2011).

To account for multiple observations from each panellist, the logistic regressions were specified with standard errors clustered at the panellist level.

Albright, J.J., Mair, P.: Does the number of parties to place affect the placement of parties? Results from an expert survey experiment. Elect. Stud. 30, 858–864 (2011)

Armstrong, J.S.: How to make better forecasts and decisions: avoid face-to-face meetings. Foresight 5, 3–8 (2006)

Benoit, K., Laver, M., Mikhaylov, S.: Treating words as data with error: uncertainty in text statements of policy positions. Am. J. Polit. Sci. 53, 495–513 (2009)CrossRef

Bolger, F., Wright, G.: Reliability and validity in expert judgment. In: Wright, G., Bolger, F. (eds.) Expertise and Decision Support, pp. 47–76. Plenum, London (1992)CrossRef

Booker, J.M., Meyer, M.A.: Sources and effects of interexpert correlation: an empirical study. IEEE Trans. Syst. Man Cybern. 18, 135–142 (1988)CrossRef

Brambor, T., Clark, W.R., Golder, M.: Understanding interaction models: improving empirical analyses. Polit. Anal. 14, 63–82 (2006)CrossRef

Braun, D., Mikhaylov, S., Schmitt, H.: European Parliament Election Study 2009, Manifesto Study [electronic file]. GESIS-Leibniz Institute for the Social Sciences [distributor] ZA5057 (2010). doi:10.4232/1.10204

Budge, I.: Expert judgements of party policy positions: uses and limitations in political research. Eur. J. Polit. Res. 37, 103–113 (2000)

Conti, N., Memoli, V.: The multi-faceted nature of party-based Euroscepticism. Acta Polit. 47, 91–112 (2012)CrossRef

Curini, L.: Experts’ political preferences and their impact on ideological bias: an unfolding analysis based on a Benoit-Laver expert survey. Party Polit. 16, 299–321 (2010)CrossRef

Dalkey, N.C.: Toward a theory of group estimation. In: Linstone, H.A., Turoff, M. (eds.) The Delphi Method: Techniques and Applications, pp. 236–261. Addison Wesley, Reading (1975)

Dalkey, N.C., Helmer, O.: An experimental application of the Delphi method to the use of experts. Manag. Sci. 9, 458–467 (1963)CrossRef

Dalton, R.J., Beck, P.A., Huckfeldt, R.: Partisan cues and the media: information flows in the 1992 presidential election. Am. Polit. Sci. Rev. 92, 111–126 (1998)CrossRef

de Lange, S.L.: A new winning formula? The programmatic appeal of the radical right. Party Polit. 13, 411–435 (2007)CrossRef

De Swaan, A.: An empirical model of coalition formation as an n-person game of policy distance minimization. In: Groenings, S., Kelley, E.W., Leiserson, M. (eds.) The Study of Coalition Behavior, pp. 424–444. Holt, Rinehart and Winston. New York (1970)

Dinas, E., Gemenis, K.: Measuring parties’ ideological positions with manifesto data: a critical evaluation of the competing methods. Party Polit. 16, 427–450 (2010)CrossRef

Djouvas, C., Gemenis, K., Mendez, F., Wheatley, J.: SmartCoding: an online platform for estimating parties’ policy positions. Unpublished manuscript, available at www.preferencematcher.org (2014)

Dolezal, M.: The design of the study: the distinguishing characteristics of our approach. In: Kriesi, H., Grande, E., Lachat, R., Dolezal, M., Bornschier, S., Frey, T. (eds.) West European Politics in the Age of Globalization, pp. 53–74. Cambridge University Press, Cambridge (2008)

Dolezal, M., Ennser-Jedenastik, L., Müller, W.C., Winkler, A.K.: How parties compete for votes: a test of saliency theory. Eur. J. Polit. Res. 53, 57–76 (2014)CrossRef

Downs, A.: An Economic Theory of Democracy. Harper and Brothers, New York (1957)

Einhorn, H.J.: Expert judgment: some necessary conditions and an example. J. Appl. Psychol. 59, 562–571 (1974)CrossRef

Ferrell, W.R.: Combining individual judgments. In: Wright, G. (ed.) Behavioral Decision Making, pp. 111–145. Plenum Press, London (1985)CrossRef

Garzia, D., Marschall, S. (eds.): Matching Voters with Parties and Candidates. ECPR Press, Colchester (2014)

Gemenis, K.: Proxy documents as a source of measurement error in the Comparative Manifestos Project. Elect. Stud. 31, 594–604 (2012a)

Gemenis, K.: A new approach for estimating parties’ positions in voting advice applications. Paper presented at the XXVI congress of the Italian Political Science Association, Rome, 13–15 September 2012b.

Gemenis, K.: What to do (and not to do) with the Comparative Manifestos Project data. Polit. Stud. 61(S1), 23–43 (2013a)

Gemenis, K.: Estimating parties’ policy positions through voting advice applications: some methodological considerations. Acta Polit. 48, 268–295 (2013b)CrossRef

Gemenis, K., Dinas, E.: Confrontation still? Examining parties’ policy positions in Greece. Comp. Eur. Polit. 8, 179–201 (2010)CrossRef

Gemenis, K., Van Ham, C.: Comparing methods for estimating parties’ positions in voting advice applications. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 33–47. ECPR Press, Colchester (2014)

Golder, M., Stramski, J.: Ideological congruence and electoral institutions. Am. J. Polit. Sci. 54, 90–106 (2010)CrossRef

Grimmer, J., Stewart, B.M.: Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Polit. Anal. 21, 267–297 (2013)CrossRef

Gudbrandsen, F.: Partisan influence on immigration: the case of Norway. Scand. Polit. Stud. 33, 248–270 (2010)CrossRef

Hawkins, K.A.: Is Chávez populist? Measuring populist discourse in comparative perspective. Comp. Polit. Stud. 42, 1040–1067 (2009)CrossRef

Janda, K.: Political Parties: A Cross-National Survey. Free Press, New York (1980)

Jones Jr, R.J.: The state of presidential election forecasting: the 2004 experience. Int. J. Forecast. 24, 310–321 (2008)CrossRef

Katakis, I., Tsapatsoulis, N., Mendez, F., Triga, V., Djouvas, C.: Social voting advice applications: definitions, challenges, datasets and evaluation. IEEE Trans. Syst. Man Cybern. 44, 1039–1052 (2014)

Krippendorff, K.: Content Analysis: An Introduction to its Methodology, 2nd edn. Sage, Thousand Oaks (2004)

Krouwel, A., van Elfrinkhof, A.: Combining strengths of methods of party positioning to counter their weaknesses: the development of a new methodology to calibrate parties on issues and ideological dimensions. Qual. Quant. 48, 1455–1472 (2014)CrossRef

Krouwel, A., Vitiello, T., Wall, M.: The practicalities of issuing vote advice: a new methodology for profiling and matching. Int. J. Electron. Gov. 5, 223–243 (2012)CrossRef

Krouwel, A., Vitiello, T., Wall, M.: Helping voters to vote? Dynamics created by VAAs in election campaigns between parties, voters and media. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 67–78. ECPR Press, Colchester (2014)

Lacewell, O.P., Werner, A.: Coder training: key to enhancing reliability and validity. In: Volkens, A., Bara, J., Budge, I., McDonald, I., Klingemann, H.-D. (eds.) Mapping Policy Preferences from Texts III, pp. 169–194. Oxford University Press, Oxford (2013)

Lau, L.J., Frey, B.: Ideology, public approval, and government behavior. Public Choice 10, 21–40 (1971)CrossRef

Laver, M.: Position and salience in the policies of political actors. In: Laver, M. (ed.) Estimating the Policy Position of Political Actors, pp. 66–75. Routledge, London (2001)

Lewis, J.B., King, G.: No evidence on directional vs. proximity voting. Polit. Anal. 8, 21–33 (1999)CrossRef

Lowe, W.: Understanding wordscores. Polit. Anal. 16, 356–371 (2008)CrossRef

Mair, P.: Searching for the positions of political actors: a review of approaches and a critical evaluation of expert surveys. In: Laver, M. (ed.) Estimating the Policy Position of Political Actors, pp. 10–30. Routledge, London (2001)

Meyer, M.A., Booker, J.M.: Eliciting and Analyzing Expert Judgment: A Practical Guide. Academic Press, London (1991)

Mikhaylov, S., Laver, M., Benoit, K.: Coder reliability and misclassification in the human coding of party manifestos. Polit. Anal. 20, 78–91 (2012)CrossRef

Mulgrave, N.W., Ducanis, A.J.: Propensity to change responses in a Delphi round as a function of dogmatism. In: Linstone, H.A., Turoff, M. (eds.) The Delphi Method: Techniques and Applications, pp. 288–290. Addison Wesley, Reading (1975)

Müller, W.C., Strøm, K.: Coalition governance in Western Europe: an introduction. In: Müller, W.C., Strøm, K. (eds.) Coalition Governments in Western Europe, pp., pp. 1–31. Oxford University Press, Oxford (2003)

Mumpower, J.L., Stewart, T.R.: Expert judgement and expert disagreement. Think. Reason. 2, 191–212 (1996)CrossRef

Neuendorf, K.A.: The Content Analysis Guidebook. Sage, Thousand Oaks (2002)

Odmalm, P.: Party competition and positions on immigration: strategic advantages and spatial locations. Comp. Eur. Polit. 10, 1–22 (2012)CrossRef

Parenté, F.J., Anderson-Parenté, J.K.: Delphi inquiry systems. In: Wright, G., Ayton, P. (eds.) Judgmental Forecasting, pp. 129–156. Wiley, New York (1987)

Pellikaan, H., Van der Meer, T., de Lange, S.L.: The road from a depoliticized to a centrifugal democracy. Acta Polit. 38, 23–49 (2003)CrossRef

Riffe, D., Lacy, S., Fico, F.G.: Analyzing Media Messages: Using Quantitative Content Analysis in Research. Lawrence Erlbaum Associates, Mahwah (2005)

Rowe, G.: Perspectives on expertise in the aggregation of judgments. In: Wright, G., Bolger, F. (eds.) Expertise and Decision Support, pp. 155–180. Plenum, New York (1992)CrossRef

Rowe, G., Wright, G.: The Delphi technique as a forecasting tool: issues and analysis. Int. J. Forecast. 15, 353–375 (1999)CrossRef

Ruedin, D.: Obtaining party positions on immigration in Switzerland: comparing different methods. Swiss Polit. Sci. Rev. 19, 84–105 (2013)CrossRef

Scheibe, M., Skutsch, M., Schofer, J.: Experiments in Delphi methodology. In: Linstone, H.A., Turoff, M. (eds.) The Delphi Method: Techniques and Applications, pp. 262–287. Addison Wesley, Reading (1975)

Shanteau, J., Hall, B.: What does it mean when experts disagree. In: Salas, E., Klein, G. (eds.) Linking Expertise and Naturalistic Decision Making, pp. 229–244. Lawrence Erlbaum Associates, Mahwah (2001)

Shikano, S.: Estimating ideological positions of political parties using a deliberative expert survey. Paper presented at the voting advice applications workshop, University of Twente, The Netherlands, 15 November 2013.

Shikano, S., Busemeyer, M., Djouvas, C., Elff, M., Gemenis, K., Mendez, F., Wheatley, J.: ParteieNavi: voting advice application data for the 2013 German federal election [electronic file]. GESIS-Leibniz Institute for the Social Sciences [distributor] (2014). doi:10.7802/68

Steenbergen, M.R., Marks, G.: Evaluating expert judgments. Eur. J. Polit. Res. 46, 347–366 (2007)CrossRef

Stokes, D.E.: Spatial models of party competition. Am. Polit. Sci. Rev. 57, 368–377 (1963)CrossRef

Tilley, J., Wlezien, C.: Does political information matter? An experimental test relating to party positions on Europe. Polit. Stud. 56, 56–214 (2008)CrossRef

Trechsel, A.H., Mair, P.: When parties (also) position themselves: an introduction to the EU Profiler. J. Inf. Technol. Polit. 8, 1–20 (2011)CrossRef

Tversky, A., Kahneman, D.: Judgment under uncertainty: heuristics and biases. Science 185, 1124–1131 (1974)CrossRef

Volkens, A.: Strengths and weaknesses of approaches to measuring policy positions of parties. Elect. Stud. 26, 108–120 (2007)CrossRef

Walter, A.S., van der Brug, W., van Praag, P.: When the stakes are high: party competition and negative campaigning. Comp. Polit. Stud. 47, 550–573 (2014)

Zulianello, M.: Analyzing party competition through the comparative manifesto data: some theoretical and methodological considerations. Qual. Quant. 48, 1723–1737 (2014)CrossRef

Title: An iterative expert survey approach for estimating parties’ policy positions
Author: Kostas Gemenis
Publication date: 01-11-2015
Publisher: Springer Netherlands
Published in: Quality & Quantity / Issue 6/2015
Print ISSN: 0033-5177
Electronic ISSN: 1573-7845
DOI: https://doi.org/10.1007/s11135-014-0109-5

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 6/2015

Voters’ preference diversity, concepts of agreement and Condorcet’s paradox

The lack of probability culture in Italy. Toward an international comparative research program

Organizational decision-maker bias supports merger wave formation: demonstration with logical formalization

Towards a method for synthesizing diverse evidence using hypotheses as common language

Qualitative variations: the sources of divergent qualitative methodological approaches

Theory of fuzzy non homogeneous Markov systems with fuzzy states

Premium Partner