Explaining variation in the competitiveness of U.S. Senate elections, 1922–2004

In a mature democratic system, political competition involves party and candidate entry across elections; competition for votes in elections; and competition within legislatures (or among different governments) between elections. Here we limit ourselves to the second of these aspects. Even though we do so, the literature on just this part of the general topic is substantial. There is a large literature starting with Downs (1957) that deals with how electoral competition affects voter turnout (e.g., Filer and Kenny 1980; Cox and Munger 1989; Endersby et al. 2002; Franklin 2004), and there are a smaller number of papers on how variation in competitiveness influences public policy, some addressing this question using data from U.S. states (e.g., Levitt and Poterba 1999 and Besley et al. 2010) or using data from other countries (Remmer and Wibbels 2000; Ferris et al. 2008; Galasso and Nannicini 2011). Also, regression discontinuity studies have focused on the candidates in close elections (e.g., Lee et al. 2004; Ferreira and Gyourko 2007; Pettersson-Lidbom 2008; Caughey and Sekhon 2011). Political competition has also been studied as a dependent variable. One component of this literature, also inspired by Downs, models candidate success by studying platform positioning relative to the median voter (e.g., Ansolabehere et al. 2001; Austen-Smith 1987; Gerber and Lewis 2004; Robertson 1976; Roemer 2001).

A large literature has sought to understand competitiveness in individual contests by focusing on factors such as candidate experience and funding (e.g., Berry and Canon 1993; Jacobson 2006; Francis and Kenny 2000). In our pursuit of institutional and structural factors that affect competitiveness, rather than individual characteristics of competitors, we follow Taagepera and Shugart (1989) and Taagepera (1999, 2007), among others.

Our work on the Senate can be thought of as complementary to the body of work inspired by Mayhew (1974), referred to as the 'vanishing marginals' literature, that is concerned with trends in the competitiveness of contests for the US House of Representatives. For example, Abramowitz et al. (2006) support the finding that redistricting is not the primary villain in reducing political competition in the US House, but argue that increasing polarization has led to a larger number of non-competitive seats. Jacobson (2006) reviews work seeking to account for the decline of competition in the House. Merrill et al. (forthcoming) is conceptually related to our explanatory conundrum; they ask, how, with no change in electoral rules, levels of mean party polarization in the U.S. House can dramatically vary over time. See also the discussion of the importance of constraints on candidate locations in Grofman (1987).

In virtually all models of party formation, when voters differ ideologically, parties form in a way that separates them along ideological lines. See, for example, Aldrich (1983), Gomberg et al. (2004), Moon (2004), Poutvaara (2003), and Snyder and Ting (2002).

The 17th Amendment to the Constitution providing for direct election of Senators was approved in 1913, and elections were phased in over the next several years. By 1922, all elections to the Senate were direct.

We note that averaging over time is essential in the present context because a big win in any one election does not establish a systematic pattern of deviation from a regime of close contests. (Buchler (2009) has argued that a big win from time to time may be required to incentivize political parties in a democratic system.) A Methodological Appendix, available on Stanley Winer's website: www.​carleton.​ca/​~winers, contains the rationale for averaging over 12-year periods and the robustness of our results to alternative periodization strategies.

Later we disaggregate data by region as well as by time period. Inspection of the data on which Table 1 is based shows that variation in competitiveness occurs across states within a given time period as well as over time.

While changes in the sizes of state populations (Hibbing and Brandes 1983) might be a candidate for such a theory, it cannot account for the types of variation we observe in the Table 1 data. Another explanatory candidate is political realignment. Below and in the on-line Appendix we consider how to take realignment related effects into account; this does not affect our conclusions.

In contrast with a too simple reading of Downs (1957), in two-party plurality competition convergence of candidates to the median in the constituency should not be expected. Rather, for a plethora of reasons (see e.g., Calvert and Isaac 1981; Palfrey 1984, 1989; Poole and Rosenthal 1984; Wuffle et al. 1989; Grofman et al. 1990; Coughlin 1992; Ansolabehere et al. 2001; Grofman 2004; Adams et al. 2005; Callander 2005; Schofield and Sened 2006; and Hussey and Zaller 2011), we should expect to see candidates located somewhere between their national party median and the constituency median. It is useful to recall that Downs himself (1957, p. 98) saw problems with convergence as an electoral strategy: "Citizens see little point in voting if all choices are identical, so differences between platforms must be created to entice voters to the polls."

On this point see, for example, Jacobsen (2006) and Masket (2009). On the particular importance of out-of-state campaign financing, the need for which constrains local candidates, see Sorauf (1992) and Marshall (1998). For data concerning out-of-state financing from 2000 to 2010, see www.​opensecrets.​org.

In recent elections, newspaper articles have associated this sort of constraint on the policy location of Republican candidates with the rise of the “Tea Party,” and the subsequent loss of some moderate Republicans to primary challenges.

Callander (2005), for example, explores a setting in which national parties compete simultaneously for multiple districts of heterogeneous voters. In his national party model, the party selects the platform that will be espoused by all of its candidates in every district. In this case, he shows that the national parties select different platforms in equilibrium.

For example, in the 109th Congress, in the US House of Representatives DW-Nominate range is about 2. If we look only at the 235 Republicans, DW-nominate scores ranged from 0.136 to 1.0281, with a mean of 0.518 and a standard deviation of 0.167.

Conceptual models that predict this stylized fact are, however, hard to find in the existing literature. One exception is recent work by Polborn (2014).

This prediction has received some empirical support. See for example, Gerber and Morton (1998) and others cited below.

Empirically we have seen that divergence is established by the DW-Nominate data. Theoretically, one can look at models by Coleman (1971, 1972), Aranson and Ordeshook (1972) and many others since.

We take the Palfrey model only as illustrative of models predicting divergence and we are, for present purposes, agnostic about the exact causes of candidate divergence. Concerning the Palfrey model, see also Greenberg and Shepsle (1987), Palfrey (1989), and Osborne (1993).

Complexities caused by competition over multiple dimensions are beyond the scope of this paper.

Cf. Esiasson (1999). To escape the constraint, the local candidate may substitute personal funds for party funds, or rely on a cult of personality. In the first case, we should expect that candidates facing a relatively hostile national party will tend to self-finance to a greater extent, and in the extreme we may see independent, largely self-financed candidates such as Ross Perot. Testing these predictions is left for future research.

Cf. Carsey and Harden (2010) and Harden and Carsey (2012).

We could consider an increase in the dispersion of voters in selected states where the constraint is binding, with analogous qualitative consequences.

In Callander's National Party-type model, heterogeneity of the electorate is a key determinant of national party platform divergence and of the number of parties in equilibrium. But he is not concerned with explaining variation in the competitiveness of local elections illustrated in Table 1. Froman (1963), Bond (1983) and Koetzle (1998) examine the impact of the heterogeneity on competition empirically, but the evidence in this literature is, on our reading, inconsistent.

Effects of changes in within-party heterogeneity should be distinguished from effects of a change in between-party caucus polarization, even though the two are empirically correlated. If between-party caucus polarization reflects generally the degree of polarization in, or dispersion of, preferences in local electorates represented by d in the simple model, the result of that development can be predicted using the Mixed Influence Model: a decrease (increase) in between-party caucus polarization will result in fewer (more) highly competitive races, because existing party-specific constraints are then binding in more (fewer) places.

We assume that in open primaries there is no strategic voting in which Republicans, for example, vote for the Democratic candidate who would be the weakest candidate in the general election.

Snyder and Ting (2011, p. 783) assert: “[with] primaries, all but the most moderate constituencies are dominated by one party.”

See www.​carleton.​ca/​~winers.

Our use of 12-year terms is defended in the on-line Appendix. We recall that averaging is also essential because a big win or loss in any one election does not signal systematic variation in competitiveness.

For example, in the 2004 presidential election, Massachusetts had the smallest share of votes cast for G. W. Bush (0.37) and was coded as 0.02 (=1/50). Utah cast the largest share of votes for Bush (0.73) and was coded as 1(=50/50). Using a rank relative to the number of states included in that election’s returns also allows us to control for omission of observations due to third party success and to control for a gain of observations because of the entry of Alaska and Hawaii into the union in 1959.

On the empirical importance of homogeneity in the party caucus for party cohesion and strength in Congress and its implications for public expenditure, see Winer et al. (2008).

In 1954–1955, the wording was “voters receive ballots of [one party] or [both parties].”

The same caveat applies, of course, to Republican candidates in very liberal states.

To construct the open seat sample, election races with incumbents are excluded from the raw data, and the 12-year averages are then recomputed.

The procedure of choosing break points that minimize a mean squared error is similar to that used in maximum likelihood estimation.

The literature on primaries and platform selection is complex, as one of the reviewers has stressed. For example, Kanthak and Morton (2001) argue that this relationship is non-linear. Furthermore, open primaries may lead more than two candidates to enter the primary, making it difficult to assess the impact on platform selection, and thus on competition. The theoretical ambiguity in the effect of open primaries on political competition certainly may a role in accounting for the relatively weak empirical support for open primaries having some effect on political competition. See also Snyder and Ting (2011).

The large national media surveys done by Gallop, CBS/NY Times, and so on are less desirable for measuring a state’s variance in voter ideology because they don’t extend back very far and rely on a crude three-category (conservative, moderate, liberal) measure of voter preferences. Robert Erikson, in an October 1, 2009 email, laments that with these data “opinion heterogeneity is very difficult to impossible to measure.” We note also that there are other survey measures of state level ideology used, for example, in Harden and Carsey (2012) in work that also points to the importance of heterogeneity of local electorates, though it is not concerned with explaining competitiveness. But these surveys are available for only a few, more recent years. Here see also Ensley (2012, pp. 43–44).

And we earlier noted that a further reason for constraints on local candidates is that swing voters, exposed to messages from the national party, may not believe that a Republican candidate, for example, is as liberal as he or she claims to be.

We note that Gerber and Lewis (2004) have explored the role of heterogeneity in voter preferences. They find that the influence of a district’s median ideology on the voting record of the district’s elected officials is greater in homogeneous (low variance of voter preferences) districts than in heterogeneous ones. This result is difficult to interpret because the regression also contains a variable for party ideology that is appropriate in a divergent platform regression but not in one based on the median voter model.