At the Nexus of Observational and Experimental Research: Theory, Specification, and Analysis of Experiments with Heterogeneous Treatment Effects

Hypotheses most commonly examine whether, on average, the treatment is zero for the units in the study. Tests of the sharp null (that the treatment effect is zero for all units in the study) are beyond the scope of this paper (for an instructive distinction between the two, see Keele 2015).

Second-generation questions include not only questions of moderation but also of mediation. Through mediation analysis, researchers attempt to establish the degree to which a given treatment operates through a particular causal mechanism (i.e., see Baron and Kenny 1986 for a clear exposition). Questions of moderation can examine how observed, longstanding predispositions that are not affected by the treatment (and are often measured pre-treatment) causally affect the magnitude of the treatment effect. Questions of mediation, in contrast, may be approached by examining how the treatment affects some observed variable that is measured post-treatment (i.e., a psychological attitude) that subsequently affects the key outcome variable of interest; or questions of mediation may be approached by embedding the proposed mediator in the experimental design. Mediation analysis has received its own fair share of attention from political scientists recently (i.e., Imai et al. 2011, 2013; Bullock et al. 2010) and lies outside the scope of this paper.

Bivariate OLS (regressing y on T) returns an unbiased estimate of the average treatment effect, even if the actual treatment effect varies across units. We will return to this point later when we walk through our scenarios.

The journals are: American Journal of Political Science, American Political Science Review, Journal of Politics, Political Behavior, and Political Psychology.

Values do not sum to 100% as only the most frequently used moderators are included.

Researchers have developed various other techniques to uncover heterogeneous treatment effects. Two-way (m)ANOVA and regression analysis yield identical hypothesis tests, but (m)ANOVA does not as easily lend itself to discussions of the substantive magnitude of heterogeneous treatment effects or the calculation of marginal effects (or statistical tests therein) across heterogeneous groups. Split-sample approaches involve comparisons of means or regression. Comparison of means across different subgroups allows for a more flexible specification for the moderation effect in comparison to the standard OLS method which assumes linearity (Hainmueller et al. 2016). However, it does not allow the researcher to simultaneously model multiple moderating relationships as OLS does. Split-sample regression (that is, splitting the sample by categories of the proposed moderator) can conveniently display differential treatment effects across different groups of units, which can be informative, but does not as conveniently facilitate statistical tests of different treatment effects compared with pooled-sample regression with an interaction (Kam and Franzese 2007). Moreover, splitting the sample by groups is non-intuitive for continuous variables (Green and Kern 2012), and it quickly becomes inefficient as the number of groups increases (and the number of units in each group decreases) (Horiuchi et al. 2007). In addition to these regression-based approaches, Horiuchi et al. (2007) and Green and Kern (2012) each provide Bayesian approaches to estimating heterogeneous treatment effects, but these approaches are well beyond the scope of this paper, and neither directly addresses our core concern regarding confounders.

Mutz (2011) notes that “A central advantage of experiments is that analysis is typically simpler and more elegant than for observational data” (p. 126).

We also note that these moderators are typically (and preferably) measured prior to the delivery of the stimulus or they are assumed (or already shown) to be stable across time and thereby uncontaminated by the experiment. Such decisions about when to measure moderators are beyond the scope of this paper; for a lively discussion on this topic, see Huber and Lapinski (2006) and Mendelberg (2008).

For ease of exposition, our example only has one treatment group. The lessons easily extend to multiple treatment groups.

It is worth noting an important difference between our framework and that of Freedman (2008). Freedman (2008) specifies a finite sample of units in the study that constitute the population of interest and that are not randomly drawn from a superpopulation. He also specifies that the only source of variability is the randomization to treatment. Using this finite-sample approach, Freedman (2008) notes that regression analysis on experimental data with covariate adjustment is biased in small samples that have unequally sized treatment and control groups. We note that Freedman’s (2008) finite-sample view diverges from our approach to analysis of experimental data. Green (2009), “Rarely do political scientists take the view that their experimental subjects exhaust the population of observations that is governed by a given set of parameters” (21). We use an infinite sampling approach, considering that any given dataset is but one of an infinite set of possible instantiations. With this framework, we can examine the properties of estimators in expectation. It is also worth mentioning that even if one adopts Freedman’s finite-sample approach, Green (2009) notes that Freedman’s warnings about bias are “negligible” for studies that are drawn from populations greater than 20—a floor condition that nearly all experiments in political science exceed.

In finite samples, imbalanced treatment and control conditions may introduce their own set of issues such as “tyranny of the majority” which can be addressed through weighted least squares (Lin 2013).

ε_i ~ N(0,4).

In the 1000 simulations, the average R², or squared correlation between Xβ (the true signal) and y (signal plus noise), ranges from 0.20 (in Case 1) to 0.49 (Case 3). This is a reasonable degree of noisiness for social-science data.

See Note 13 on how our approach differs from that of Freedman (2008).

Note that we also include the interaction between M and T, but that is irrelevant to the point here.

On this point, the Variance Inflation Factor can inform the degree to which a particular covariate (or its interaction) imposes an efficiency cost. There is a degrees-of-freedom cost to the inclusion of covariates but such degrees of freedom issues are trivial in most experiments, where N ≫ k.

If the covariate (M) is mean-centered before being interacted with the treatment, then a regression of y on T, mean-centered M, and their interaction will allow b_T to reveal the treatment effect at the covariate’s sample mean, not at the value where M happens to equal zero, and some of the issues of bias and inefficiency in b_T are remedied. Since, in practice, most applied researchers do not mean-center their covariates, we have not done so here.

In Eq. (2), δy/δM = β_M + β_MTT = 0.5 + 0.5*T. β_M- reflects the effect of M when T = 0, and β_MT indicates how the effect of M changes when T = 1. When the interaction term is omitted, β_M returns the weighted average effect of M, which in this scenario is the average of 0.5 (the effect of M in the control condition) and 1 (the effect of M in the treatment condition). Note the average estimate of β_M is ~0.75 in Table 2, model (b).

The extent of the efficiency loss for b_MT depends upon the correlation between the two moderators (M and Z). In supplemental simulations (not shown), we varied Corr(M,Z) from 0 to 0.9. When the two moderators were not all correlated (Corr(M, Z) = 0), model(d) yields 20 simulations (of 1000) where the null hypothesis (β_MT = 0) would be rejected, and model(e) yields exactly the same number where the null hypothesis would be rejected. As Corr(M, Z) rises, the likelihood of making a Type II error (rejecting the null hypothesis that b_MT = 0 when there actually is an effect) rises. In the text, where Corr(M, Z) = 0.5, and 97.7% of the 1000 simulations correctly reject the null when model (d) is used, whereas only 92.7% of the 1000 simulations correctly reject the null when model(e) is used. At the extreme, where Corr(M,Z) = 0.9, 97.2% of the 1000 simulations correctly reject the null when model(d) is used, whereas only 40.2% of the simulations correctly reject the null when model(e) is used. In short, when an extraneous interaction between Z and T is included, the more correlated Z and M are, the more likely it is that the researcher will incorrectly infer a lack of a moderating effect due to the collinearity-induced inflation of the standard errors.

The estimates for β_M and β_Z in model (b) are the weighted averages for the effects of M in the control and treatment (~0.75) and the effects of Z in the control and treatment (~1.125).

If the conditional expectation function is not linear, regression adjustment with covariates can be mildly biased, though consistent. We thank an anonymous reviewer for this point.

As with OLS, matching will have no effect on the estimate of the ATE as long as the matching scheme is unrelated to assignment to treatment.

See Imai et al. (2008) for discussion of the Sample Average Treatment Effect.