Randomization Inference in the Regression Discontinuity Design: An Application to Party Advantages in the U.S. Senate

2.1 Hypothesizing the randomization mechanism

The first task in applying randomization inference to the RD design is to choose a randomization mechanism for ZW0 that is assumed to describe the data generating process that places units on either side of the threshold. A natural starting place for a setting in which Zi is an individual-level variable (as opposed to a group-level characteristic) assumes Zi is a Bernoulli random variable with parameter π. In this case, the probability distribution of ZW0 is given by Pr(ZW0=z)=πz′1(1−π)(1−z)′1, for all vectors z in ΩW0, which in this case consists of the 2nW0 possible vectors of zeros and ones, where nW0 is the number of units in W0 and 1 is a conformable vector of ones. This randomization distribution is fully determined up to the value π, which is typically unknown in the context of RD applications. A natural choice for π would be πˆ=ZW0′1/nW0, the fraction of units within the window with scores exceeding the threshold.^[5]

While the simplicity of this Bernoulli mechanism is attractive, a practical disadvantage is that it results in a positive probability of all units in the window being assigned to the same group. An alternative mechanism that avoids this problem, and is also likely to apply in settings where Zi is an individual-level variable, is a random allocation rule or “fixed-margins randomization” in which the number of units within the window assigned to treatment is fixed at mW0. Under this mechanism, ΩW0 consists of the nW0mW0 possible nW0-vectors with mW0 ones and nW0−mW0 zeros. The probability distribution is Pr(ZW0=z)=nW0mW0−1, for all z∈ΩW0.

When Zi is a group-level variable, or where additional variables are known to affect the probability of treatment, other mechanisms approximating a block-randomized or stratified design will be more appropriate.

2.2 Test of no effect

Having chosen an appropriate randomization mechanism, we can test the sharp null hypothesis of no treatment effect under Assumption 1. No treatment effect means observed outcomes are fixed regardless of the realization of ZW0. Under this null hypothesis, potential outcomes are not a function of treatment status inside W0; that is, yi(z)=yi for all i within the window and for all z∈ΩW0, where yi is a fixed scalar. The distribution of any test statistic T(ZW0,yW0) is known, since it depends only on the known distribution of ZW0, and yW0, the fixed vector of observed responses. The test thus consists of computing a significance level for the observed value of the test statistic. The one-sided significance level is simply the sum of the probabilities of assignment vectors z leading to values of T(z,yW0) at least as large as the observed value T˜, that is, Pr(T(ZW0,yW0)≥T˜)=∑z∈ΩW01(T(z,yW0)≥T˜)⋅Pr(ZW0=z), where Pr(ZW0=z) follows the assumed randomization mechanism.

Any test statistic may be used, including difference-in-means, the Kolmogorov–Smirnov test statistic, and difference-in-quantiles. While in typical cases the significance level of the test may be approximated when a large number of units is available, randomization-based inference remains valid (given Assumption 1) even for a small number of units. This feature is particularly important in the RD design where the number of units within W0 is likely to be small.

2.3 Confidence intervals and point estimates

While the test of no treatment effect is often an important starting place, and appealing for the minimal assumptions it relies on, in most applications we would like to construct confidence intervals and point estimates of treatment effects. This requires additional assumptions. The next assumption we introduce is that of no interference between units.

Assumption 2: Local stable unit treatment value assumption. For all i with Ri∈W0: if zi=z˜i then yi(zW0)=yi(z˜W0).

This assumption means that unit i’s potential outcome depends only on zi, which, together with Assumption 1, allows us to write potential outcomes simply as yi(0) and yi(1) for units in W0. Assumptions 1–2 enable us to characterize the effects of treatment through inference on the distribution or quantiles of the population of nW0 potential outcomes in W0, {yi(z):Ri∈W0}, as in Rosenbaum ([14], Chapter 5). The goal is to construct a confidence interval [a(q),b(q)] that covers with at least some specified probability the q-quantile of {yi(1):Ri∈W0}, denoted Q1(q), which is simply the ⌈q×nW0⌉-th order statistic of {yi(1):Ri∈W0} for units within the window W0, and a similar confidence interval for Q0(q). The confidence interval for Q1(q) consists of the observed treated values x above the threshold (but in the window) such that the hypothesis H0:Q1(q)=x is not rejected by a test of at most some specified size. The test statistic is J(x)=ZW0′1(YW0≤x), the number of units above the threshold whose outcomes are less than or equal to x, and has distribution Pr(J(x)=j)=(⌈q×nW0⌉j)(nW0−⌈q×nW0⌉mW0−j)/(nW0mW0) under a fixed-margins randomization mechanism where mW0 denotes the number of treated units inside W0. Inference on the quantile treatment effect Q1(q)−Q0(q) can be based on confidence regions for Q1(q) and Q0(q).

Point estimates and potentially shorter confidence intervals for the treatment effect can be obtained at the cost of a parametric model for the treatment effect. A simple (albeit restrictive) model that is commonly used is the constant treatment effect model described below.

Assumption 3: Local constant treatment effect model. For all i with Ri∈W0: yi(1)=yi(0)+τ, for some τ∈ℝ.

Under Assumptions 1–3, and hypothesizing a value τ=τ0 for the treatment effect, the adjusted responses, Yi−τ0Zi=yi(0), are constant under alternative realizations of ZW0. Thus, under this model, a test of the hypothesis τ=τ0 proceeds exactly as the test of the sharp null discussed above, except that now the adjusted responses are used in place of the raw responses. The test statistic is therefore T(ZW0,YW0−τ0ZW0), and the significance level is computed as before. Confidence intervals for the treatment effect can be found by finding all values τ0 such that the test τ=τ0 is not rejected, and Hodges–Lehmann-type point estimates can also be constructed finding the value of τ0 such that the observed test statistic T(ZW0,YW0−τ0ZW0) equals its expectation under the null hypothesis.

We discuss this constant and additive treatment effect model because it allows us to illustrate how confidence intervals can be easily derived by inverting hypothesis tests about a treatment effect parameter. But there is nothing in the randomization inference framework that we have adopted that necessitates Assumption 3. This assumption can be easily generalized to allow for non-constant treatment effects, such as Tobit or attributable effects (see Rosenbaum [15], Chapter 2). Indeed, the technique of constructing adjusted potential outcomes and inverting hypothesis tests of the sharp null hypothesis is general and allows for arbitrarily heterogeneous models of treatment effects. Furthermore, the confidence intervals for quantile treatment effects described above do not require a parametric treatment effect model.

3.1 Implementation

Implementing the procedure proposed above requires three choices: (i) a test statistic, (ii) the minimum sample sizes (j0,min, j1,min), and (iii) a testing procedure and associated significance level α. We discuss here how these choices affect our window selector, and give guidelines for researchers who wish to use this procedure in empirical applications.

4.1 RD design in U.S. Senate elections: two estimands of party advantage

Our application of the RD design to U.S. Senate elections focuses on two specific estimands that capture local electoral advantages and disadvantages at the party level. The first estimand, which we call the incumbent-party advantage, focuses on the effect of the Democratic party winning a Senate seat on its vote share in the following election for that seat. The other estimand, which we call the opposite-party advantage following Alesina et al. [27], is unrelated to the traditional concept of the incumbency advantage and reveals the disadvantages faced by the party that tries to win the second seat in a state’s Senate delegation. Establishing whether the opposite-party advantage exists has been of central importance to theories of split-party Senate delegations, and there are different explanations of why it may arise.^[9]

Both estimands, formally defined in terms of potential outcomes below, are derived from applying an RD design to the staggered structure of Senate elections, which we now describe briefly. Term length in the U.S. Senate is 6 years and there are 100 seats. These Senate seats are divided into three classes of roughly equal size (Class I, Class II and Class III), and every 2 years only the seats in one class are up for election. As a result, the terms are staggered: in every general election, which occurs every 2 years, only one third of Senate seats are up for election. Each state elects two senators in different classes to serve a 6-year term in popular statewide elections. Since its two senators belong to different classes, each state has Senate elections separated by alternating 2-year and 4-year intervals. Moreover, in any pair of consecutive elections, each election is for a different senate seat – that is, for a seat in a different class.^[10]

Following Butler and Butler [16], we apply the RD design in the U.S. Senate analogously to its previous applications in the U.S. House, comparing states where the Democratic party barely won election t to states where the Democratic party barely lost. But in the Senate, the staggered structure of terms adds a layer of variability that allows us to both study party advantages and validate our design in more depth than would be possible in a non-staggered legislature such as the House. Using t, t+1 and t+2 to denote three successive elections, the staggered structure of the Senate implies that the incumbent elected at t, if he or she decides to run for reelection, will be on the ballot at t+2, but not at t+1, when the Senate election will be for the other seat in the state. As summarized in Table 1, this staggered structure leads to two different research designs analyzing two separate effects.

The first design (Design I) focuses on the effect of party P’s barely winning at t on its vote share at t+2, the second election after election t, and defines the first RD estimand we study. As illustrated in the third row of Table 1, in Design I elections t and t+2 are for the same Senate seat, and this incumbent-party effect captures the added vote share received by the Democratic party due to having won (barely) the seat’s previous election. The second research design (Design II), illustrated in the second row of Table 1, allows us to analyze the effect of party P’s barely winning election t on the vote share it receives in election t+1 for the state’s other seat, when the incumbent candidate elected at t is, by construction, not contesting the election. Thus, Design II defines the second RD estimand, the opposite-party advantage, which will be negative when the party of the sitting senator (elected at t) is at a disadvantage relative to the opposing party in the election for the other seat (which occurs at t+1).

Using the notation introduced in Section 2, we consider two estimands defined by Designs I and II. We define the treatment indicator as Zit=1(Rit≥r0) and the potential outcomes in elections t+2 and t+1, respectively, as yit+2(Zit) and yit+1(Zit).^[11] Thus, the incumbent-party advantage for an individual state i is defined as τiIP=yit+2(1)−yit+2(0) and the opposite-party advantage as τiOP=yit+1(1)−yit+1(0). Our randomization inference approach to RD offers hypothesis testing and point-type estimators (e.g., Hodges–Lehmann) of these parameters, possibly restricted by a treatment effect model, for the units in the window W0 where local randomization holds.

5.1 Selecting the window

We selected our window using the method based on predetermined covariates presented in Section 3. The largest window we considered was [−100,100], covering the entire support of our running variable. Based on power considerations discussed above, the minimum window we considered was [−0.5,0.5], because within this window there are 9 and 14 outcome observations to the left and right of the cutoff, respectively, and we wanted to set j0,min and j1,min to be approximately equal to 10. Using our notation in Section 3, this means we set [R(j0,min),R(j1,min)]=[−0.50,0.50] and [R(1),R(n)]=[−100,100]. We analyzed all symmetric windows around the cutoff between [−0.5,0.5] and [−100,100] in increments of 0.125 percentage points. In each window, we performed randomization-based tests of the sharp null hypothesis of no treatment effect for each of eight predetermined covariates: state-level Democratic percentage of the vote in the past presidential election, state population, Democratic percentage of the vote in the t−1 Senate election, Democratic percentage of the vote in the t−2 Senate election, indicator for Democratic victory in the t−1 Senate election, indicator for Democratic victory in the t−2 Senate election, indicator for open Senate seat at t, indicator for midterm (non-presidential) election at t and indicator for whether the president of the U.S. at t is Democratic. As discussed above, we set α=0.15, and use the difference-in-means as the test statistic in our randomization-based tests. These tests (and similar tests for the outcomes presented below) are based on 10,000 simulations of the randomization distribution of ZW0 assuming a fixed-margins assignment mechanism. For each window, we chose the minimum p-value across these eight covariates.

Figure 1 summarizes graphically the results of our window selector. For every symmetric window considered (x-axis), we plot the minimum p-value found in that window (y-axis). The x-axis is the absolute value of our running variable, the Democratic margin of victory at election t, which is equivalent to the upper limit of each window considered (since we only consider symmetric windows) and ranges from 0 to 100. For example, the point 20 on the x-axis corresponds to the [−20,20] window. The figure also shows the conventional significance level of 0.05 and the significance level of 0.15 that we use for implementation. There are a few notable patterns in this figure. First, for most of the windows considered, the minimum p-value is indistinguishable from zero, which means that there is strong evidence against Assumption 1 in most of the support of our running variable. Second, the minimum p-value is above the conventional 5% significance level in very few windows (15 out of the total 797 windows considered). Third, the decrease in p-values is roughly monotonic and very rapid, suggesting that Assumption 1 is implausible except very close to the cutoff. Using α=0.15, our chosen window is [−0.75,0.75], the third smallest window we considered, since this is the largest window where the minimum p-value exceeds 15% in that window and all the windows contained in it.

Figure 1

Window selector based on predetermined covariates

Table 2 shows the minimum p-values for the first five consecutive windows we considered and also for the windows [−1.5,1.5], [−2,2], [−10,10] and [−20,20]. The minimum p-value in our chosen window is 0.2682, and the minimum p-value in the next largest window, [−0.875,0.875], is 0.0842. P-values decrease rapidly after that and, with some exceptions such as around window [−1.50,1.50], do so monotonically. Note also that had we set α=0.10, our chosen window would still have been [−0.75,0.75]. And if we had set α=0.05, our chosen window would have been [−0.875,0.875], barely larger than our final choice, which shows the steep decline of the minimum p-value as we include observations farther from the cutoff.

Our window selection procedure suggests that Assumption 1 is plausible in the window [−0.75,0.75]. Further inspection and analysis of the 38 observations in this window (23 treated and 15 control) shows that these observations are not associated in any predictable way. These electoral races are not concentrated in a particular year or geographic area: these 38 races are spread across 24 different years with no more than 3 occurring in the same year, and 26 different states with at most 4 occurring in the same state. This empirical finding further supports the idea that these observations might be treated as-if randomly assigned. Moreover, an important implication of this finding is that there is no observable clustering structure in the sample inside the window [−0.75,0.75], which in turn implies that standard randomization inference techniques are directly applicable. Finally, we also performed standard density tests for sorting and found no evidence of any systematic discrepancy between control and treatment units.^[13] Thus, below we proceed to make inferences about the treatment effects of interest under Assumption 1 in this window.

5.2 Inference within the selected window

We now show that the results obtained by conventional methods are robust to our randomization-based approach in both Design I and Design II. Randomization-based results within the window imply a sizable advantage when a party’s same seat is up for election (Design I) that is very similar to results based on conventional methods. Randomization results on outcomes when the state’s other seat is up for reelection (Design II) show a null effect, also in accordance with conventional methods. However, as we discuss below, the null opposite advantage results from Design I are sensitive to our window choice, and a significant opposite-party advantage appears in the smallest window contained within our chosen window.

Our randomization-based results include a Hodges–Lehmann estimate, a treatment effect confidence interval obtained inverting hypothesis tests based on a constant treatment effect model, a quantile treatment effect confidence interval, and a sharp null hypothesis p-value calculated as described in the window selection section above. Table 3 contrasts the party advantage estimates and tests obtained using our randomization-based framework, reported in column (3), to those obtained from two classical approaches: a 4th-order parametric fit as in Lee [9] reported in column (1), and a nonparametric local-linear regression with a triangular kernel as suggested by Imbens and Lemieux [2], using a mean-squared-error (MSE) optimal bandwidth implementation described in Calonico et al. [8], reported in column (2). For both approaches, we show conventional confidence intervals; for the local linear regression results, we also show the robust confidence intervals developed by Calonico et al. [8], since the MSE optimal bandwidth is too large for conventional confidence intervals to be valid.^[14] Panel A presents results for Design I on the incumbent-party advantage, in which the outcome is the Democratic vote share in election t+2. Panel B presents results for Design II on the opposite-party advantage, in which the outcome is the Democratic vote share in election t+1. Our randomization-based results are calculated in the window [−0.75,0.75] chosen above. Note that, as mentioned above, there is no need for clustering in our window, nor is clustering empirically possible.

The point estimates in the first row of Panel A show an estimated incumbent-party effect of around 7 to 9 percentage points for standard RD methods and 9 percentage points for the randomization-based approach. These estimates are highly significant (p-values for all three approaches fall well below conventional levels) and point to a substantial advantage to the incumbent party when the party’s seat is up for re-election. In other words, our randomization-based approach shows that the results obtained with standard methods are remarkably robust: a local or global approximation that uses hundreds of observations far away from the cutoff yields an incumbent-party advantage that is roughly equivalent to the one estimated with the 38 races decided by three quarters of a percentage point or less. This robustness is illustrated in the top panel of Figure 2. Figure 2(a) displays the fit of the Democratic Vote Share at t+2 from a local linear regression on either side of the optimal bandwidth and shows a clear jump at the cutoff of roughly 7.4 percentage points (dots are binned means). Figure 2(b) on the right displays the mean of the Democratic Vote Share at t+2 on either side of our chosen [−0.75,0.75] window (dots are individual data points) and shows a similar (slightly larger) positive jump at the cutoff.

In our data-driven window, estimates of the opposite-party party advantage also appear robust to the method of estimation employed. In Panel B, estimates on Democratic Vote Share at t+1 based on conventional methods show very small, statistically insignificant effects of around 0.64 to 0.35 in columns (1) and (2). These standard methods of inference for RD are therefore unable to reject the hypothesis of a null effect, and would suggest that, contrary to balancing and constituency-based theories, there is no opposite-party advantage in U.S. Senate elections. Our randomization-based approach, presented in column (3) of Panel B, arrives at a similar conclusion, finding a negative point estimate but a sharp null p-value above 0.80 and a 95% confidence interval for a constant treatment effect that ranges roughly between –8 and 5. Similarly, the 95% confidence intervals for the 25th and 75th quantile treatment effects are roughly centered around zero and are consistent with a null opposite-party advantage.

These results are illustrated in the bottom row of Figure 2, where Figure 2(c) and 2(d) are analogous to Figure 2(a) and 2(b), respectively. The effect of winning an election by 0.75% appears roughly equivalent to the effect estimated by standard methods. In our randomization-based window, the mean of the control group is slightly larger than the mean of the treatment group, but as shown in Table 3 we do not find statistically significant evidence of an opposite-party advantage.

Taken together, our results provide interesting evidence about party-level electoral advantages in the U.S. Senate. First, our results show that there is a strong and robust incumbent-party effect, with the party that barely wins a Senate seat at t receiving on average seven to nine additional percentage points in the following election for that seat. Second, our randomization-based approach confirms the previous finding of Butler and Butler [16], according to which there is no opposite-party advantage in the U.S. Senate. As we show below, however, and in contrast to the incumbent-party advantage results, the opposite-party advantage result is sensitive to our window choice and becomes large and significant as predicted by theory inside a smaller window.

Figure 2

RD design in U.S. Senate elections, 1914–2010 – standard local-linear approach vs. randomization-based approach

5.3 Sensitivity of results to window choice and test statistics

We study the sensitivity of our results to two choices: the window size and the test statistic used to conduct our tests. First, we replicate the randomization-based analysis presented above for different windows, both larger and smaller than our chosen [−0.75,0.75] window. We consider one smaller window, [−0.5,0.5], and two larger windows, [−1.0,1.0] and [−2.0,2.0]. We note that, given the results in Table 2, we do not believe that Assumption 1 is plausible in windows larger than [−0.75,0.75] and we therefore would not interpret a change in results in larger windows as evidence against our chosen window. Nonetheless, it is valuable to know if our findings would continue to hold even under departures from Assumption 1 in larger windows. This observation, however, does not apply when considering smaller windows contained in [−0.75,0.75], since if Assumption 1 holds inside our chosen window, it also must hold in all windows contained in it. Thus, analyzing the smaller window [−0.5,0.5] can provide evidence on whether there is heterogeneity in the results found in the originally chosen window.

Second, we perform the test of the sharp null using different test statistics. Under Assumption 1, there is no relationship between the outcome and the score on either side of the threshold within W0. In this situation, performing randomization-based tests using the difference-in-means as a test statistic should yield the same results as using other test statistics that allow for a relationship between the conditional regression function and the score. This suggests using different test statistics in the same window as a robustness check. Let the window considered be [wl,wr] and recall that the cutoff is r0. In a similar spirit to conventional parametric and nonparametric RD methods, we consider two different additional test-statistics: the difference in the predicted values Yˆi from two regressions of Yi on Ri−r0 on either side of the cutoff evaluated at Ri=r0, and the difference in the predicted values Yˆi from two regressions of Yi on Ri−r0 and (Ri−r0)2 on either side of the cutoff evaluated at Ri=r0. Below, we call the p-values based on these test statistics “p-value linear” and “p-value quadratic”, respectively.

Table 4 presents the results from our sensitivity analysis. Panel A shows results for Democratic Vote Share at t+2 (Design I), and Panel B for Democratic Vote Share at t+1 (Design II). For each panel, we reproduce the results in our chosen [−0.75,0.75] window and show results for the three additional windows mentioned above: [−0.5,0.5], [−1.0,1.0] and [−2.0,2.0]. All results are calculated as in Table 3. The “p-value diffmeans” is equivalent to the p-value reported in Table 3, which corresponds to a test of the sharp null hypothesis based on the difference-in-means test statistic. The two additional p-values reported correspond to a test of the sharp-null hypothesis based on the two additional test statistics described above. All p-values less than or equal to 0.05 are shown in bold in the table.

There are important differences between our two outcomes. The results in Design I (Panel A) are robust to the choice of the test statistic in the originally chosen [−0.75,0.75] window and in the smaller [−0.50,0.50] window. The results are also insensitive to increasing the window, as seen in the last two columns of Panel A. In contrast, the null results found in Design II seem more fragile. First, the sharp null hypothesis is rejected in some larger windows when alternative test statistics are considered. Second, in our chosen window, the sharp null hypothesis is rejected with the linear regression test statistic, but not with the quadratic regression test statistic. As we showed before in Table 3 and reproduce in Table 4, this does not translate into a statistically significant constant or quantile treatment effect – all confidence intervals are roughly centered around zero. An interesting phenomenon that might explain this pattern occurs when we consider the smaller [−0.5,0.5] window. In this window, the point estimate and confidence intervals show a negative effect and provide support for the opposite-party advantage hypothesis. The Hodges–Lehmann point estimate is about –8 percentage points, more than a 10-fold increase in absolute value with respect to the conventional estimates, and we reject the sharp null hypothesis of no effect at the 5% level with two of the three different test statistics considered. Our randomization-based confidence interval of the constant treatment effect ranges from –16.66 to –0.08, ruling out a non-negative effect. The confidence interval for the 25th quantile treatment effect also excludes zero and again provides support for the opposite-party advantage.

To investigate this issue further, Figure 3 plots the empirical cumulative distribution functions (ECDF) of our two outcomes in two different windows: the small [−0.5,0.5] window and the window defined by [−0.75,−0.50)∪(0.5,0.75]. The union of these two windows is our chosen [−0.75,0.75] window. Figure 3(a) shows that for Democratic Vote Share t+2, the ECDF of the treatment group is shifted to the right of the ECDF of the control group everywhere in both windows, showing that the treated quantiles are larger than the control quantiles. Since the treated outcome dominates the control outcome in both windows, combining the observations into our chosen window produces the robust incumbent-party advantage results that we see in the first two columns of Table 4.

In contrast, for Democratic Vote Share t+1, the outcome in Design I, the smaller [−0.5,0.5] window exhibits a very different pattern from the [−0.75,−0.50)∪(0.5,0.75] window. The left plot in Figure 3(b) shows that the ECDF of the control group is shifted to the right of the ECDF of the treatment group everywhere, showing support for the negative effect (opposite-party advantage) reported in the first column of Table 4. But the right plot in Figure 3(b) shows that this situation reverses in the window [−0.75,−0.50)∪(0.5,0.75], where treated quantiles are larger than control quantiles almost everywhere. The combination of the observations in both windows is what produces the null effects in our chosen [−0.75,0.75] window. In sum, the results in [−0.5,0.5] suggest some support for the opposite-party advantage and show that our chosen window combines possibly heterogeneous treatment effects for Vote Share t+1 (but not for Vote Share t+2).

All in all, our sensitivity and robustness analysis in this section shows that the incumbent-party advantage results are robust but our opposite-party advantage results are more fragile and suggest some avenues for future research.

Figure 3

Empirical CDFs of outcomes for treated and control in different windows – U.S. Senate elections, 1914–2010. (a) Democratic vote share at t + 2. (b) Democratic vote share at t + 1

6.1 Fuzzy RD with possibly weak instruments

In the sharp RD design, treatment assignment is equal to Zi=1(Ri≥r0), and treatment assignment is equal to actual treatment status. In the fuzzy design, treatment status Di, with observations collected in n-vector D as above, is not completely determined by placement relative to r0, so Di may differ from Zi. Our framework extends directly to the fuzzy RD designs, offering a robust inference alternative to the traditional approaches when the instrument (i.e., the relationship between Di and Zi) is regarded as “weak”.

Let di(r) be unit i’s potential treatment status when the vector of scores is R=r. Similarly, we let yi(r,d) be unit i’s potential outcome when the vector of scores is R=r and the treatment status vector is D=d. Observed treatment status and outcomes are Di=di(R) and Yi=yi(R,D). This generalization leads to a framework analogous to an experiment with non-compliance, where Zi is used as an instrument for Di and randomization-based inferences are based on the distribution of Zi. Assumption 1 generalizes as follows.

Assumption 1′: Local randomized experiment. There exists a neighborhood W0=[r_,rˉ] with r_<r0<rˉ such that for all i with Ri∈W0:

(a)FRi|Ri∈W0(r)=F(r), and

(b)di(r)=di(zW0) and yi(r,d)=yi(zW0,dW0) for all r,d.

This assumption permits testing the null hypothesis of no effect exactly as described above, although the interpretation of the test differs, as now it can only be considered a test of no effect of treatment among those units whose potential treatment status di(zW0) varies with zW0. Constructing confidence intervals and point estimates in the fuzzy design requires generalizing Assumption 2 and introducing an additional assumption.

Assumption 2′: Local SUTVA (LSUTVA). For all i with Ri∈W0:

(a)If zi=z˜i, then di(zW0)=di(z˜W0), and

(b)If zi=z˜i and di=d˜i, then yi(zW0,dW0)=yi(z˜W0,d˜W0).

Assumption 6: Local exclusion restriction. For all i with Ri∈W0: yi(z,d)=yi(z˜,d) for all (z,z˜) and for all d.

Assumption 6 means potential responses depend on placement with respect to the threshold only through its effect on treatment status. Under assumptions 1′−2′ and Assumption 6, we can write potential responses within the window as yi(z,d)=yi(di). Furthermore, under the constant treatment effect model in Assumption 3, estimation and inference proceeds exactly as before, but defining the adjusted responses as YW0−τ0DW0. Inference on quantiles in the fuzzy design also requires a monotonicity assumption (e.g., Frandsen et al. [33]).

Fuzzy RD designs are local versions of the usual instrumental variables (IV) model and thus concerns about weak instruments may arise in this context as well [34]. Our randomization inference framework, however, circumvents this concern because it enables us to conduct exact finite-sample inference, as discussed in Imbens and Rosenbaum [10] for the usual IV setting. Therefore, our framework also offers an alternative, robust inference approach for fuzzy RD designs under possibly weak instruments.

6.2 Discrete and multiple running variables

Another feature of our framework is that it can handle RD settings where the running variable is not univariate and continuous. Our results provide an alternative inference approach when the running variable is discrete or has mass points in its support (see, for example Lee and Card [35]). While conventional, nonparametric smoothing techniques are usually unable to handle this case without appropriate technical modifications, our randomization inference approach applies immediately to this case and offers researchers a fully data-driven approach for inference when the running variable is not continuously distributed. Similarly, our approach extends naturally to settings where multiple running variables are present (see, e.g., Keele and Titiunik [36] and references therein). For example, in geographic RD designs, which involve two running variables, Keele et al. [37] discuss how the methodological framework introduced herein can be used to conduct inference employing geographic RD variation.

6.3 Matching and parametric modeling

Conventional approaches to RD employ continuity of the running variable and large-sample approximations, and typically do not emphasize the role of covariates and parametric modeling, relying instead on nonparametric smoothing techniques local to the discontinuity. However, in practice, researchers often incorporate covariates and employ parametric models in a “small” neighborhood around the cutoff when conducting inference. Our framework gives a formal justification (i.e., “local randomization”) and an alternative inference approach (i.e., randomization inference) for this common empirical practice. For example, our approach can be used to justify (finite-sample exact) inference in RD contexts using panel or longitudinal data, specifying nonlinear models or relying on flexible “matching” on covariates techniques. For a recent example of such an approach, see Keele et al. [37].

6.4 Sensitivity analysis and related techniques

In the context of randomization-based inference, a useful tool to asses the plausibility of the results is a sensitivity analysis that considers how the results vary under deviations from the randomization assumption. Rosenbaum [14, 15] provides details of such an approach when the treatment is assumed to be randomly assigned conditionally on covariates. Under a randomization-type assumption, the probability of receiving treatment is equal for treated and control units; a sensitivity analysis proposes a model for the odds of receiving treatment and allows the probability of receiving treatment to differ between groups and recalculates the p-values, confidence intervals or point estimates of interest. The analysis asks whether small departures from the randomization-type assumption would alter the conclusions from the study. If, for example, small differences in the probability of receiving treatment between treatment and control units lead to markedly different conclusions (i.e., if the null hypothesis of no effect is initially rejected but then ceases to be rejected), then we conclude that the results are sensitive and appropriately temper our confidence in the results. This kind of analysis could be directly applied in our context inside W0. In this window, our assumption is that the probability of receiving treatment is equal for all units (and that we can estimate such probability); thus, a sensitivity analysis of this type could be applied directly to establish whether our conclusions survive under progressively different probabilities of receiving treatment for treated and control units inside W0.

6.5 Connection to standard RD setup

Our finite-sample RD inference framework may be regarded as an alternative approximation to the conventional RD identifying conditions in Hahn et al. [5]. This section defines a large-sample identification framework similar to the conventional one and discusses its connection to the finite-sample Assumption 1.

In the conventional RD setup, individuals have random potential outcomes Yi(r,d) which depend on the value of a running variable, r∈ℝ, and treatment status d∈{0,1}. The observed outcome is Yi≡Yi(Ri,Di), and identification is achieved by imposing continuity, near the cutoff r0, on E[Yi(r,d)|Ri=r] or FYi(r,d)|Ri=r(y)=Pr[Yi(r,d)≤y|Ri=r]. Consider the following alternative identifying condition.

Assumption 7: Conventional RD assumption. For all d∈{0,1} and i=1,2,⋯,n:

(a)Ri is continuously distributed,

(b)Yi(r,d) is (a.s.) Lipschitz continuous in r at r0,

(c)FYi(r0,d)|Ri=r(y)=Pr[Yi(r0,d)≤y|Ri=r] is Lipschitz continuous in r at r0.

These conditions are very similar to those in Hahn et al. [5] and other (large-sample type) approaches to RD. The main difference is that we require continuity of potential outcome functions, as opposed to just continuity of the conditional expectation or distribution of potential outcomes. Continuity of the potential outcome functions rules out knife-edge cases where confounding differences in potential outcomes at the threshold (that is, discontinuities in Yi(r,d)) exactly offset sorting in the running variable at the threshold so that the conditional expectation of potential outcomes is still continuous at the threshold. In ruling out this knife-edge case, our condition is technically stronger, but arguably not stronger in substance, than conventional identifying conditions.

The conventional RD approach approximates the conditional distribution of outcomes near the threshold as locally linear and relies on large-sample asymptotics for inference. Our approach proposes an alternative local constant approximation and uses finite-sample inference techniques. The local linear approximation may be more accurate than local constant farther from the threshold but the large-sample sample approximations may be poor. The local constant approximation will likely be appropriate only very near the threshold, but the inference will remain valid for small samples. The following suggests that our finite-sample condition in Assumption 1 can be seen as an approximation obtained from the more conventional RD identifying conditions given in Assumption 7, with an approximation error that is controlled by the window width.

Result 1: connection between RD frameworks. Suppose Assumption 7 holds. Then:

(i)FRi|Ri∈[r_,rˉ],Yi(r0,d)=y(r)=FRi|Ri∈[r_,rˉ](r)+Oas(rˉ−r_), and

(ii)Yi(r,d)=Yi(r0,d)+Oas(rˉ−r_).

Part (i) of this result says that the running variable is approximately independent of potential outcomes near the threshold, or, in the finite-sample framework where potential outcomes are fixed, each unit’s running variable has approximately the same distribution (under i.i.d. sampling). This corresponds to part (a) of Assumption 1 (Local Randomization) and gives a formal connection between the usual RD framework and our randomization-inference framework. Similarly, part (ii) implies that potential outcomes depend approximately on treatment status only near the threshold r0, as assumed in Assumption 1(b).