Does the rule matter? A comparison of preference elicitation methods and voting rules based on data from an Austrian regional parliamentary election in 2019

Elections are among the most important societal activities in democracies. The use of the “correct” electoral rule is, therefore, of huge importance, because, in general, the choice of the rule matters [see, e.g., Riker (1986), Kaminski (1999), Evci and Kaminski (2021)]. In this paper, we provide an empirical analysis of the regional parliamentary elections 2019 in the Austrian region of Styria. In line with many empirical findings in the literature, we provide additional empirical evidence that the choice of the voting rule has an impact on the outcome of the election. In addition, we analyze the consistency of preference statements under different voting rules and consider the aspect of strategic voting in the context of the above-mentioned elections.

One of the first significant studies on the impact of electoral systems on election outcomes has been provided by Duverger (1951). In particular, Duverger hypothesizes that electoral systems based on plurality lead to two-party systems, whereas rules based on proportional representation tend to create many-party systems. His explanation of how the number of parties in a country depends on a country’s electoral system has later been called “Duverger’s Law” in the literature. A further empirical approach can be found in Rae (1971), who carefully studies different electoral systems and their impact on parties. Inspired by this approach, Lijphart (1994) systematically investigates the political consequences of electoral systems in 27 democracies trying to reduce the confusion surrounding electoral systems.

Given the theoretical observation and empirical evidence that electoral rules do indeed matter, electoral engineering gained a certain amount of importance. Riker [see Riker (1986), (1988)] provides examples of political manipulation by means of changing the used voting method in order to gain an advantage, including the Third Force in France. Taagepera and Shugart (1989) analyze examples of unwarranted changes in voting rules and their impact on the respective countries. In addition, they provide certain guidelines for justified changes in electoral systems. Empirical investigations of the impact of electoral engineering have been studied, e.g., in Poland (Kaminski, 1999, 2002), where the design of voting rules for the post-communist area has been very illuminating. In particular, it has been shown that the impact of voting rules on the electoral outcome has not been perfectly understood by the relevant agents, leading to unintended disadvantages (i.e., a lower number of seats) as a result of changing the voting system. More recently, Evci and Kaminski (2021) provide a similar analysis about unintended political consequences of political engineering for Turkey. McCune and McCune (2022), however, provide an empirical study of American single-winner elections and conclude that in these elections, apart from the plurality rule, the choice of voting method rarely matters. Note that in contrast to that study, we are concerned with a regional parliamentary election using proportional representation, and we are not exclusively concerned with the party ranked first in the social outcome under a specific voting rule. The contribution of institutional factors and institutional choices to the transition and consolidation of new democracies in the developing world has been studied by Elklit and Reynolds (2002). Finally, there also have been studies examining the influence of linguistic aspects on voter behavior (Gerber et al., 2016) and party-related primacy effects because a candidate is listed first on a ballot (Flis & Kaminski, 2022).

A slightly different empirical approach can be found in the literature on behavioral social choice, which provides a comparison of the theoretical behavior of rational agents with how real decision makers in fact behave. Contributions such as Regenwetter and Tsetlin (2004), Regenwetter et al. (2006), Regenwetter et al. (2007) or Popov et al. (2014), analyze the consequences of theoretical results for the practical use of social choice rules, raising doubt about the empirical importance of certain negative results such as those on Condorcet cycles.

In recent years many experimental and survey studies have been conducted in connection with real-world elections. Perhaps the most relevant studies related to this paper are those by Baujard et al. (2021), Baujard et al. (2018), Baujard et al. (2014) on French elections, by Roescu (2014) on a Romanian election, by Alos-Ferrer and Granić (2012) on German elections and Darmann et al. (2017, 2019) on the 2015 regional parliamentary elections in Styria. In addition, there exists a substantial empirical and experimental literature in the field of political science which is concerned with elections especially in relation to strategic voting [see, e.g., Blais et al. (2005), Spenkuch (2015), Stephenson et al. (2018), and Blais and Degan (2019)].

In this paper, following our analysis of the 2015 parliamentary elections in the Austrian region of Styria (Darmann et al., 2017), we add an analysis of the regional elections in the year 2019. In contrast to the 2015 elections with eight parties on the ballot, in the 2019 elections only six well-known parties were running (with all of them eventually gaining seats in the parliament). Our research is based on an exit-poll survey—specifically on election day, in front of some randomly chosen polling stations, voters were asked to respond to various questions concerned with different ways to state their preferences by filling out a questionnaire. Those preferences have then been used in various ways.

First, we examine the social rankings of the parties under different voting rules and find that, for some parties especially, their positions vary considerably depending on the voting rule. Second, we use the data to analyze the properties and consistency of different versions of voters’ preferences as elicited by our survey. Third, we examine how the voting rules allow voters to behave strategically, leading to a discrepancy between elicited preferences and reported votes.

The paper is structured as follows: Sect. 2 introduces the survey design. Section 3 provides the hypothetical results under various different voting rules. This is followed by an analysis of the voters’ consistency in stating their preferences in different forms in Sect. 4. Section 5 discusses strategic behavior in the election and, finally, Sect. 6 concludes the paper.

The data for this survey study were collected during election day, Sunday 24 November 2019, for the regional parliamentary election in the Austrian region of Styria.¹ Compared to other elections and previous elections in the region, the selected election was particularly interesting. There were exactly six parties running, all of them well-known and viable, i.e., with good chances to get seats in the parliament. Actually, all parties eventually did get enough votes to enter the parliament. To get a glimpse of voters’ preferences and feelings as closely connected as possible to the actual act of voting, we undertook an exit poll. In front of nine randomly chosen voting stations in the city of Graz, voters, when they left the polling station, were approached by our staff and invited to provide written and anonymous information about their preferences over the six parties running in the election.² The 937 voters who participated in the exit poll were given a two-page questionnaire that asked them to report which party they voted for in the actual election, to rank the six parties in order of preference, to indicate which parties they “approve”, to evaluate the parties on various scales, and to answer some demographic questions. A translated version of the questionnaire can be found in the Appendix.

When eliciting preference information, we intentionally avoided reference to any voting rule for two reasons: First, explaining such rules on the spot would be too complicated to ensure a deeper understanding. Second, a possible strategic behavior based on the voting rules seems implausible given that no detailed explanation of the voting rules were given and no real election (and therefore no consequences) is related to the participants’ preferences. Hence, we assume to have received the voters’ sincere preferences.

Our survey included about 5.5% of the voters in the nine randomly chosen polling stations. Given that the voters participated voluntarily in the survey, the raw data obviously has a certain participation bias. This can be seen in comparing the actual voting results with the declared votes by the participants. We observe a clear “liberal bias”, i.e., voters of more leftist/liberal parties such as Greens, KPÖ and NEOS were much more willing to participate in the survey. On the contrary, declared support for rightist parties, in particular for the FPÖ, was rather small, indicating that they either refused to participate, or were rather “ashamed” to declare their rightist attitude (although answers were anonymous and only in writing). We correct that bias by the use of weights; the weights were determined by dividing the shares for each party in the official result in those nine polling stations by the share the same party received in the exit poll.³ For example, the SPÖ had a share of \(18.5\%\) of the votes in the official result and received a support of \(13.76\%\) in the exit poll. Therefore participants supporting the SPÖ were under-represented in the exit poll leading to a \(\frac{18.50}{13.76}=1.34\). If we compare the calculated weights, we observe that, in addition to the SPÖ, the ÖVP and the FPÖ also have weights larger than one and are therefore under-represented. The other parties are over-represented in the survey leading to weights smaller than one. Table 1 provides an overview over official and declared voting results in the nine polling stations.⁴

Given that our survey made no reference to voting rules, there was no incentive for strategic behavior. Hence, we assume all the preference information to be sincere. As will be shown later, strategic behavior does, however, occur in real elections [see, e.g., Blais and Degan (2017) and the literature cited therein]. This can be observed from the fact that, besides their preferences, participants were also asked to communicate for whom they had actually voted (and will be discussed in more detail in Sect. 5). When comparing the impact of different voting rules on the outcome, however, we will use exclusively the weighted (and sincere) preference data from the survey. Moreover, the current regional election was a parliamentary election. Hence, the goal of the parties was not purely to receive votes but to ensure that seats in the parliament are achieved. However, because it is not clear how to distribute seats based on any election outcome using more information than just top-ranked alternatives, we will stick to “social” rankings of the parties based on the hypothetically received votes only.

Obviously, our study faces certain limitations. In particular, given that we use the voters’ sincere preferences in determining outcomes under different voting rules, this does not lead to simulations of outcomes of actual electoral systems, because any strategic behavior based on a specific voting rule can of course not be observed. Hence, the comparison takes place purely under the hypothetical assumption of elections without any strategic behavior.

Our survey study is closely related to Darmann et al. (2017), in which the 2015 regional parliamentary elections in Styria were investigated. The current survey is different in three respects. First, in the 2015 elections, eight parties were running, including all six parties from the 2019 elections and, additionally, two very small parties (that, together, received less than 2% of total votes). Second, the 2015 survey included a slightly different set of preference elicitation methods, leading to some differences in the set of voting rules analyzed. Third, the survey in 2015 includes voters from all over Styria, whereas the current survey only includes voters from the city of Graz (the largest of four voting districts). Interestingly, the political situation changed quite drastically between the elections of 2015 and 2019. In particular, the overall election results showed massive changes with ÖVP (\(+7.60\) percentage points), Greens (\(+5.40\)), KPÖ (\(+1.77\)) and NEOS (\(+2.73\)) gaining, and SPÖ (\(-6.27\)) and FPÖ (\(-9.27\)) losing. Comparing the official results restricted to the nine polling stations used for the 2019 survey, we also observe a huge difference as displayed in Table 2 (the 2015 percentages are based only on the votes for the six parties stated).

Reasons for the huge differences can be seen in various political developments between 2015 and 2019. The national government changed to a coalition of ÖVP and Greens (putting the Greens in the government for the first time). Moreover, the KPÖ has been consistently very successful in the city of Graz and the NEOS experienced significant support on a national level. All of that can partly explain those changes. Hence, it seems that comparisons between the survey studies in 2015 and 2019 are rather difficult to establish.

Concerning the hypothetical elections results, the 2015 study had a very consistent winner (SPÖ) and second place party (ÖVP). Also, the current study, as will be shown below, provided a very consistent outcome with the Greens winning in eight out of nine hypothetical elections. Based on the definitions in Darmann et al. (2017), the current survey illustrates the impact of voting rules using more elaborate preference information on medium and polarizing parties. Intuitively, a medium party is acceptable to a large proportion of the voters but induces strong positive or negative views only in small groups, whereas a polarizing party induces strong views (both, negatively and positively) in a large group of voters. As will be shown in this section, medium parties (such as the NEOS) will experience a positive impact from voting rules using more elaborate preference information, whereas polarizing parties (such as the FPÖ, but also the ÖVP) rather experience a negative impact.⁵

In the following we want to introduce each of the nine voting rules which we applied on the preferences stated by the participants of the exit poll.⁶ We selected those rules because of their practical or theoretical importance. The order of the voting rules is to a certain extent based on the complexity of the rule itself and the preference information needed.

In an interesting contribution, Baujard et al. (2018) analyze the consistency among voting rules focusing on different types of approval voting. In particular, they show that under different voting rules, the announced preferences might, because of strategic considerations, be different. Obviously, our setting does not allow for such a comparison given that potentially used voting rules have not been communicated to the participants. However, given that the preference data available to us could be seen as being based on sincere voting behavior, we are able to analyze the consistency among different preference elicitation methods. It seems clear that ranking all of the parties is considerably more demanding than just voting for one party. Hence, the question arises whether voters would actually be able to provide such extended preference information. During the exit poll, voters were asked to which extent providing a full ranking of the parties was difficult or easy. Surprisingly, more than 75% of the voters who responded to that question considered this to be (rather) easy. Although the same question was not explicitly asked about the different forms of evaluative voting, at least for approval voting and 3-scale evaluative voting, the difficulty of providing that information does not seem to exceed the difficulty of providing a full ranking. This can also be seen in the percentages of voters that provided the necessary preference information for the various voting rules to be applied, presented in Table 12.⁸

Although our sample of voters is not representative, there is a clear indication that the intuitive differences in the difficulty of providing the varying preference information are similar to those perceived by the actual voters.

However, it is of considerable interest whether this apparent simplicity of stating different types of preferences also leads to consistent preferences, i.e., whether the preference stated in a more complex form can be translated into a preference of some less complex type. A simple first check could be seen in comparing approval voting with the Borda rule in the sense that an approved party should never receive a lower Borda score (i.e., be lower ranked) than a disapproved party. If that was not the case, this would imply inconsistent behavior by the voters. In a similar manner we compare approval voting with 3-scale evaluative voting and evaluative voting, i.e., we verify whether approved parties were given evaluations at least as high as disapproved ones. Table 13 shows the percentage of voters (who communicated the respective two types of preferences) that provided consistent preferences w.r.t. their approval votes. For example, \(93.78\%\) of the voters approved their top-ranked party.

Hence, if measured in this mild sense, as explained above, the voters have been very consistent at least with respect to their (more complex) preferences compared with their approval preferences. We can extend this to a comparison of the other three types of preferences, sticking to this mild comparison, i.e., only requiring that if a party a is considered at least as good as party b under preference type 1, then party a also has to be considered at least as good as party b under preference type 2. Insisting on strict preferences would drastically lower the rate of consistency and, if compared to the full rankings, where indifference could not be communicated, probably not correctly reflect the actual consistency. Table 14 provides the percentages of voters that consistently stated their preferences of the respective two types.

To conclude, the participants of the exit poll have been quite consistent in stating their preferences. Obviously, consistency was more likely to be observed when the dichotomous preference of approval voting was compared to any of the other types of preferences, and to be particularly high when extending the scale by just one degree. Most difficulties (and therefore errors) occurred when rankings and large-scale evaluations had to be provided. Here, the consistency level dropped to below two thirds. However, if one would still lower the threshold for consistency (e.g., by allowing for at most one mistake), further increases in consistency would occur.

Baujard et al. (2021) investigate, among other interesting aspects, a different kind of consistency, namely the impact of the length of the scale on evaluative voting and the use of extreme grades. They show that the distribution of the grades remains stable for different scale-lengths when reduced linearly to two classes. In addition, the use of extreme (negative) grades under different scales is very stable for polarizing parties. In our study, because of lack of respective data, a reduction to two classes does not seem appropriate. However, it is possible to analyze the consistency w.r.t. “extreme” valuations. For example, Fig. 2 shows the use of lowest grades in 3-scale EV and EV when voters did not approve a party. Because EV allowed for 41 different grades, we divided those grades into three roughly equally sized classes and used the lowest class with grades in the interval \(\left[ -20,-7\right]\) for the comparison (see Fig. 2).

Obviously, the 45-degree line indicates a perfect match between the evaluations using the two preference elicitation methods. As can be seen, the use of lowest grades (or lowest intervals in the case of EV) was very consistent. The evaluations for all of the parties lie close to the 45-degree line. Figure 2, however, also shows another interesting aspect. Large percentages of low grades in case a party is not approved indicate strong objection of the respective party. That is, parties which are far from the origin can be seen as polarizing or unpopular in the definition of Darmann et al. (2017). In the Styrian election, as seen in the figure, this is the case in particular for the FPÖ, but also to some extent for the ÖVP. In principle, the FPÖ is indeed perceived to be on the right of the political spectrum. The ÖVP has, historically, rather been seen as a popular party, i.e., one that is seen positively by a large part of society with strong support from a specific group of voters; however, in recent years it took some right-wing positions on certain political issues giving it a slightly polarizing character. On the other hand, medium parties (i.e., parties which are acceptable to a large fraction of society inducing strong views in small groups only) and popular parties would rather be close to the origin. Again this can be confirmed by looking at the NEOS and Greens in the picture. Of course, one could consider the interval for EV too large, but the picture does not significantly change if the interval is restricted to, e.g., \(\left[ -20,-15\right]\).

In addition, we could also look at the positive support and compare the different scales. Figure 3 compares the percentages of positive support under 3-scale EV and evaluations in the interval \(\left[ 7,20\right]\) under EV.

Again, the behavior of using highest grades is rather consistent given that all points are close to the 45-degree line. Concerning the types of the parties, the picture is not so clear. Very popular parties should of course be far from the origin, because voters that approve such a party should also evaluate it highly. This is indeed the case for the Greens. Medium parties, on the other hand, should not get, in general, such a strong support even if they are approved. This can be seen to hold for the NEOS and the KPÖ. On the other hand, for a polarizing party, one would expect also very strong support from those who approve this party. However, this is not the case for the FPÖ, although it can be classified as a polarizing party: roughly one quarter of the voters who approve the FPÖ do not give them the highest grades under EV and 3-scale EV.

In contrast to the negative grades, here the picture does change when the interval for EV is restricted to, e.g., \(\left[ 15,20\right]\), as shown in Fig. 4. Hence, the voters seem reluctant to give rather high evaluations for a party, which might indicate a limited level of satisfaction with the political landscape.

All parties do experience a drop in the share of highest grades in EV. This also indicates that voters do actually use the extended scale when evaluating the parties.

Finally, we can look at the voters’ behavior w.r.t. their grades distributed whenever parties have been approved or ranked highly. Table 15 provides some interesting comparisons about the voters’ preferences.⁹

The first four lines of Table 15 show the percentages of voters who approve a party (AV(1)) and evaluate them with grades as specified (e.g., EV(>0) means that the party received an evaluation larger than 0, and EV(20) indicates that the party received an evaluation of 20). Obviously, the strongest support under EV, i.e., a grade of \(+20\), is only given by roughly one third of the supporters for popular and polarizing parties such as the Greens, ÖVP and FPÖ. Medium parties, as was to be expected, such as the NEOS and the SPÖ, receive a much smaller share of strongest support. Given that the median approval rate was two parties (see Sect. 3.7), one could also check (line 5) how approved parties have been ranked by the voters, i.e., whether they were among the top two ranked parties (rank(1,2)). Again, medium and rather non-polarizing parties might have been approved even from voters that do not rank them highly. Polarizing and popular parties which are approved, are higher ranked more often. The values in line 6 of Table 15 provide some information about how intensely the voters value their top-ranked party, i.e., which percentage of those voters that have a party top-ranked also assign them the highest possible grade. There is no clear tendency visible, but it seems that between one third and one half of the voters translate a top rank into the highest grade. This is rather surprising insofar as, theoretically, under EV it could not harm to always assign the highest possible grade to the party a voter likes most. Surely, because there was no real election based on those stated preferences, we would rather expect sincere behavior, and that is what the data seem to confirm. Finally, line 7 does the opposite, i.e., it shows how many voters that had a party bottom-ranked actually gave it a grade of \(-20\). Interestingly, negative feelings have been expressed much more intensely, especially towards polarizing parties such as the FPÖ and the ÖVP, where roughly two thirds of the voters that had them bottom-ranked also assigned them a grade of \(-20\).

In addition to the previously discussed (in)consistency in a voter’s preferences, which rather depends on the complexity involved in stating one’s preferences, another discrepancy might depend on a voter’s intentional misstatement of her preferences. That is, the actual vote differs from the voter’s preferences. In terms of plurality rule, this would mean that a voter does not cast her vote for her highest ranked party. This is a widely discussed topic in the literature, usually termed strategic voting, which would deserve a paper on its own and therefore will only be briefly explained in this study for the sake of completeness.

Based on various famous theoretical contributions in political science [e.g., Duverger (1951) and Cox (1997)], recent years have shown a large increase in empirical literature on strategic voting [e.g., Blais and Degan (2017), Stephenson et al. (2018), Spenkuch (2018) and the cited literature therein]. Most of these empirical studies focus on single-winner plurality elections. However, in recent years more attention (using both surveys and lab experiments) has been given to proportional representation and mixed systems [see Blais and Degan (2019) for an overview]. Our analysis falls into the first category of survey-based approaches. One major advantage of our study is that our participants responded immediately after voting and in the same location, making the link to the actual election as close as possible. As has been pointed out before [see, e.g., Wright (1992) and Alvarez and Nagler (2000)], one major caveat can be seen in the fact that self-declaration of preferences is not without doubt. Obviously also in our data, certain announced behavior by some voters cannot be reasonably explained.

Most survey-based studies estimate the frequency of strategic voting to be within the range of 3% to 17% of the ballots cast [see Spenkuch (2013)]. Given our data, we can easily determine how many supporters of a party, i.e., voters that have this party top-ranked according to their announced preferences, did actually vote for a different party,¹⁰ i.e., behave strategically. Going into more detail, one, at least initially surprising, fact is that, under plurality rule, the winner based on the stated preferences (Greens) is different compared to the winner determined from the declared official votes (ÖVP). This is a consequence of voters not voting for their most preferred party (as stated in their ranking) and, hence, indicates strategic voting. Out of the 820 participants in the survey that responded to those questions, 113 did not vote for their top-ranked party, i.e., we observe a share of roughly 14% of the voters who voted strategically.¹¹ There could exist various different reasons for a voter to vote strategically [see Blais and Degan (2017) for a more detailed discussion], depending on all sorts of information available to the voter when she casts her vote, i.e., pre-election polls, media coverage, etc. For example, one reason for a strategic vote can be seen in trying to avoid wasting one’s vote. If a voter’s top-ranked party has no chance to enter the parliament, then a voter might vote strategically for another party. A second reason might lie in the fundamental belief that a certain party (and therefore its distinctive concerns) should be represented in the parliament. Hence, if such a party might seem close to the threshold for entering the parliament, a voter might give her vote to such a party. Third, if the actually formed coalition after the election is important for a voter, then she might vote strategically to push a party to have enough seats together with the seats of her most preferred party to form a coalition, or to support a party to increase its strength in an expected coalition.

The previous reasons should give some arguments for strategic voting being rational. However, whatever reason for strategic behavior might be assumed, it does not seem rational for a voter to vote strategically for a very low ranked party, which did occur occasionally (10% of the strategic votes were for a party ranked in fourth position or lower). Hence, we would rather see this as irrational behavior (or, simply, a mistake) and, therefore, “reasonable” strategic voting in our data is observed for about 12.5% of the voters.¹² Table 16 shows how the supporters of a party (i.e., those that had the party top-ranked) did actually vote.

We observe that supporters of the Greens and the NEOS in particular did use their votes to a much higher percentage to act strategically. The reasons could, however, have been very different. According to all pre-election polls, the Greens were certain to get enough votes to enter the parliament. In addition, it was also very unlikely that they would be part of any coalition forming the regional government. Hence, supporters of the Greens did use their votes extensively to support either parties that were rather on the edge of entering the parliament (KPÖ) or those that could potentially be part of the government (SPÖ) and were considered ideologically closer to the Greens than other parties (such as the ÖVP or the FPÖ).

On the contrary, part of the supporters of the NEOS might have had the impression that their vote could be “wasted” given the chance that they might not get enough votes to enter the parliament (as has happened in the previous elections in 2015), allocating their vote to other parties, especially those for which seats in the parliament were certain (such as the ÖVP and the Greens).

Besides the behavior of voters of the respective parties, we can also check from which supporters a party actually received its declared official votes. This is displayed in Table 17.

Table 17 nicely shows the dynamics of the strategic voting. The KPÖ received a lot of votes from supporters of the Greens, which does intuitively make sense given that in pre-election polls the KPÖ was rather on the edge of entering the parliament and the general political orientation of Greens and KPÖ seems sufficiently similar. The same is the case for the NEOS with the additional fact that, given their more medium political orientation, they also attracted substantial votes from ÖVP supporters.

However, although most of the observed strategic behavior seems reasonable, given that the analysis is based on a single event, it does not give the possibility to state more general results.

In this paper we analyzed the 2019 Styrian Parliamentary elections based on data from an exit poll immediately after the actual election. We showed that, based on the obtained data, although the winner is stable under the nine voting rules considered, the social ranking is sensitive to different voting rules. This confirms many of the empirical studies in the behavioral social choice literature that find very consistent winners, but also the theoretical influence of voting rules on voting outcomes. Following Baujard et al. (2021) we were able to compare voting behavior under different preference query methods. It was shown that the use of evaluative voting under different grading scales was rather consistent. However, a comparison of preferences in terms of rankings and evaluations based on the \(\left[ -20,20\right]\)-scale showed considerable inconsistencies. In addition, we investigated the use of evaluative voting using longer scales, being able to show that this extended freedom in the evaluation of parties is only used in a limited way. Finally, a brief discussion of strategic behavior in the actual election has been provided. Although the general size of strategic behavior is, to a certain extent, in line with previous studies and the voters’ behavior could be reasonably explained, more general conclusions do not seem easy to draw because of the fact that data exists only for a singular event and has been obtained via self-declaration of the voters. In addition to deriving more general conclusions, an interesting direction for future research, however, could consist in parliamentary seat simulations [see Flis et al. (2020) and Evci (2020)] for each electoral rule.