When to use matching and weighting or regression in instrumental variable estimation? Evidence from college proximity and returns to college

The number of available methods for causal inference has seen enormous growth in the last three decades (see Abadie and Cattaneo 2018, for a recent overview). Although progress has been tremendous, applied research does not always keep up. On the one hand, flexible semi- and non-parametric methods based on the propensity score (PS) are widely applied when estimating causal effects under the selection-on-observables assumption (e.g., see Austin and Stuart 2015; Thoemmes and Kim 2011). On the other hand, much of the literature using instrumental variable (IV) estimation to overcome bias due to unobserved factors relies on two-stage least squares (2SLS). This is despite the fact that 2SLS may yield inconsistent estimates of treatment effects when effects are heterogenous and covariates are predictive of the instrument (Abadie 2003).

This paper concentrates on the most common case in which the researcher aims to estimate causal effects of some treatment using a single binary IV. First, the paper reviews available results from the literature on implications of using standard linear-in-covariates 2SLS estimation under effect heterogeneity. Most importantly for this paper, the literature shows that 2SLS yields a ratio of conditional variance-weighted average of covariate-specific effects. If effects are indeed heterogenous and related to the PS, then 2SLS yields inconsistent estimates. Estimators based on the PS provide a consistent (Frölich 2007), readily-available and intuitive alternative. Hence, the paper briefly describes some basic IV estimators using the PS as well as the novel efficient covariate balancing approach by Heiler (2021). By re-estimating the returns to college using these approaches—exploiting college proximity as an instrument using the data by Card (1995)—the paper shows that the threat of obtaining inconsistent estimates when using 2SLS is not merely hypothetical. 2SLS yields systematically larger effect estimates than more flexible estimators based on the PS. Further inspection shows that this difference is mainly due to the implicit conditional-variance weighting performed by 2SLS.

This case study has been widely used to teach Economics students around the world about the use of IV methods to overcome bias due to unobserved confounders as well as the importance of effect heterogeneity. Moreover, the case study has been widely used in a variety of papers, see for example Tan (2006), Huber and Mellace (2015), Kitagawa (2015), Mourifié and Wan (2017), Andresen and Huber (2021), Sloczynski (2021), Sloczynski et al. (2022) and Blandhol et al. (2022). Most of these papers are concerned with instrument validity, an issue that is discussed but not of main interest in this paper. The only study known to the author that compares parametric estimators of the returns to college with more flexible estimators for this case study is Sloczynski et al. (2022). They too find sizable gaps in estimates. However, in contrast to this paper, they do not offer an explanation for this phenomenon.

The remainder of this paper is organized as follows: Sect. 2 reviews identification and estimation using IVs, Sect. 3 applies 2SLS and comparison methods based on the PS to the data. Section 4 concludes.

Assume we have an i.i.d. sample for i = 1,…,N units, where for each unit we observe some exogenous characteristics \({\text{X}}_{\text{i}}\), a binary treatment variable \({\text{D}}_{\text{i}}\), an outcome \({\text{Y}}_{\text{i}}\) and a single binary instrument \({\text{Z}}_{\text{i}}\). Furthermore, assume that there is an unobserved confounder \({\text{U}}_{\text{i}}\) that has an impact both the treatment variable \({\text{D}}_{\text{i}}\) and the outcome \({\text{Y}}_{\text{i}}\). In the language of classical least squares regression, this creates an omitted variable bias and the selection-on-observables assumption fails (Wooldridge 2010, Chap. 4). To stick to the returns-to-college example used throughout this paper, conditioning on observed characteristics such as labor market experience or region of residence is insufficient to remove bias from standard regression or matching estimates of the effects of college attendance on wages if unobserved ability has an impact on the college decision and labor market earnings (Blackburn and Neumark 1993).

Under such circumstances, one can use IV techniques to estimate causal effects by exploiting variation in the treatment variable \({\text{D}}_{\text{i}}\) through the instrument \({\text{Z}}_{\text{i}}\). When effects are heterogenous, IV methods identify local treatment effects, i.e. average effects for specific sub-populations influenced by the instrument. For this identification result to hold, the instrument needs to be exogenous, i.e. the instrument has to be as good as randomly assigned after conditioning on covariates and there must not be a direct effect of the instrument on the outcome. Moreover, the instrument must influence the treatment decision in a monotonous way. Imbens and Angrist (1994) introduce what Sloczynski (2021) calls “strong monotonicity”, which is the assumption that the instrument weakly increases or decreases the treatment probability for everyone. Under this assumption, IV methods identify the local average treatment effect (LATE, Imbens and Angrist 1994), also called the complier average causal effect (CACE), i.e. the average treatment effect of individuals who act in line with the instrument. If defiers, i.e. individuals who act in the opposite direction of compliers, exist, and one is willing to assume that the sign of the effect of the instrument on treatment is determined solely by covariates (“weak monotonicity”, Sloczynski 2021), the CACE may be recovered by averaging effects for individuals with covariate values estimated to behave in the direction of compliers. Moreover, a more general effect, the mover average causal effect (Kolésar 2013), i.e. the average treatment effect for compliers and defiers, is identified.

Using the standard potential outcomes framework (Imbens and Angrist 1994; Rubin 1974), define \({\text{D}}_{\text{i}}\left(1\right)\)and \({\text{D}}_{\text{i}}\left(0\right)\) as the potential treatment states if the unit was assigned \({\text{Z}}_{\text{i}}=1\) or \({\text{Z}}_{\text{i}}=0\). If the instrument indeed has no direct impact on the outcome, one may write potential outcomes as \({\text{Y}}_{\text{i}}\left({d}_{i}\right)\), with \({\text{Y}}_{\text{i}}\left(1\right)\)and \({Y}_{i}\left(0\right)\) being the outcomes that would be observed under treatment and without. Assuming the instrument raises the chance of receiving treatment on average, the strong monotonicity assumption implies that for compliers \({D}_{i}\left(1\right)>{D}_{i}\left(0\right)\), i.e. they receive treatment if assigned \({\text{Z}}_{\text{i}}=1\)and they do not if assigned \({\text{Z}}_{\text{i}}=0\). Based on these definitions and assumptions, the standard CACE can be written as

The MACE is defined as \({{\Delta }}^{MACE}=E\left[{Y}_{i} \left(1\right)-{Y}_{i} \left(0\right)|{D}_{i}\left(1\right) \ne {D}_{i}(0)\right]\) and can be recovered by using a reordered instrument \({\text{Z}}_{\text{i}}^{\text{R}}\), i.e. an adapted instrument which is reversed for defiers, defined as¹where \({\delta }^{D} \left({X}_{i}\right)=E\left[{D}_{i} \right(1)-{D}_{i} (0\left)\right|{X}_{i}]\) is the covariate-specific average effect of the instrument on the treatment decision and \(I(\cdot )\) is the indicator function (Sloczynski 2021).

For the empirical analysis of the paper, the exogeneity assumption is assumed to hold. Moreover, it is assumed that at least the weak monotonicity assumption holds as well. While the failure of the monotonicity assumption makes the interpretation of estimands difficult if not impossible, differences in estimates of these quantities are still interesting to inspect in order to understand the estimators’ behavior under effect heterogeneity.

For the following exposition of estimation methods, assume that strong monotonicity holds.² While Frölich (2007) shows that the CACE is non-parametrically identified under exogeneity and strong monotonicity, most applied research still uses 2SLS to estimate effects using an IV. Typically, researchers model the outcome and treatment equations as linear functions of the instrument and covariates. That is, they build regression models that look something like where it is (implicitly) assumed that slope-coefficients are constant and that \(\varepsilon _{i}^{Y}\) and \(\varepsilon _{i}^{D}\) are well-behaved error terms. The corresponding 2SLS estimator can be written as \({\widehat{{\Delta }}}_{2SLS}={\widehat{\gamma }}^{Y}/{\widehat{\gamma }}^{D}\), i.e. the ratio of the reduced form (3) OLS coefficient \({\widehat{\gamma }}^{Y}\) and the first stage (4) OLS coefficient \({\widehat{\gamma }}^{D}\) on \({Z}_{i}\). Under effect heterogeneity and standard regularity conditions, Sloczynski (2021) shows that \({\widehat{{\Delta }}}_{2SLS}\) converges to³where \({\delta }^{Y}\left({X}_{i}\right)=E\left[{Y}_{i} \right(1)-{Y}_{i} (0\left)\right|{X}_{i}, {D}_{i} \left(1\right)>{D}_{i} \left(0\right)]\)is the average covariate-specific effect of treatment on the outcome for compliers. Hence, 2SLS yields a conditional-variance weighted average of covariate-specific effects for compliers. As \(Var\left({Z}_{i}\right|{X}_{i})=P({Z}_{i}=1\left|{X}_{i}\right)(1-P\left({Z}_{i}=1|{X}_{i}\right),\)weights attain a maximum when the PS \(P\left({Z}_{i}=1|{X}_{i}\right)=0.5\) (e.g., see Angrist and Pischke 2008). An important but typically under-appreciated consequence of this weighting is that \(\text{plim}{\widehat{{\Delta }}}_{2SLS} \ne {{\Delta }}^{CACE}\) if \({\delta }^{Y}\left({X}_{i}\right)\) and \({\delta }^{D}\left({X}_{i}\right)\) depend on the PS. Depending on the correlation structure at hand, this may lead to substantial inconsistencies when using 2SLS.

As an alternative, this paper considers IV estimators of \({{\Delta }}^{CACE}\)based on the PS as derived by Frölich (2007) as well as a recent extension by Heiler (2021).⁴ These estimators do not restrict effect heterogeneity. As a consequence, they are consistent even when effects are not homogenous (Frölich 2007, and Heiler, 2021).

The IV-matching estimator based on the PS pairs up each unit from the groups defined by the instrument with one, multiple or weighted averages of units from the opposite group based on the PS in order to infer the missing counterfactuals. Let \(\widehat{{Y}_{i}\left(1\right)}\) and \(\widehat{{D}_{i}\left(1\right)}\)denote the estimated counterfactuals for the outcome and the treatment variable if the unit was assigned \({Z}_{i}=0\) as obtained by matching. Analogously, let \(\widehat{{Y}_{i}\left(0\right)}\) and \(\widehat{{D}_{i}\left(0\right)}\) be the estimated counterfactual outcome and treatment variable if the unit was assigned \({Z}_{i}=1\). Based on this definition, the IV-matching estimator can be written as

To estimate the PS, a standard logit regression is used. Moreover, kernel matching (KM) is employed as it has been shown to be among the top-performing PS-based matching methods in several simulation studies under the selection-on-observables paradigm (e.g., see Frölich 2004; Busso et al. 2014). More specifically, the matching procedure is implemented using an Epanechnikov kernel with a bandwidth chosen via weighted cross-validation (Galdo et al. 2008). To avoid extrapolation, common support is imposed via the min-max criterion by Dehejia and Wahba (1999) as is standard in the PS-based literature (Caliendo and Kopeinig 2008).

The (un-normalized) inverse probability weighting (IPW) IV-estimator can be written aswhere \({\widehat{P}}_{i}\) is an estimate of the PS \(P\left({Z}_{i}=1|{X}_{i}\right)\). IPW has been shown to be semiparametrically efficient in the IV context (Donald et al. 2014). To estimate the PS, two approaches are used. First, the same logit estimate as for KM is employed. To ensure better performance, weights of this estimator are normalized as un-normalized weights may yield unreliable results (Frölich 2004; Busso et al. 2014). Akin to KM, the min-max criterion is used to ensure common support. In the context of IPW, this sort of trimming may be even more important as IPW with PS close to zero or one may lead to invalid statistical inference when using the non-parametric bootstrap as is done for all estimators considered. See Heiler and Kazak (2021) for derivations and alternative bootstrap approaches or Sasaki and Ura (2022) for trimming based methods.

Second, as IPW methods may be overly sensitive to the specification of the estimated PS (Schafer and Kang 2008), this paper also uses the novel efficient covariate balancing (ECB) procedure by Heiler (2021) to estimate the PS. This approach specifies a loss-function tailored to the estimation of treatment effects using IVs and algorithmically minimizes covariate imbalances, leading to improved bias and variance properties in finite-samples compared to standard IPW methods (see Heiler, 2021, for details). This approach has several advantages. First, akin to IPW, ECB is semiparametrically efficient. Second, ECB is doubly-robust if covariates are specified flexibly and third, the ECB method tends to shrink the PS which may alleviate the need to implement heuristic trimming approaches such as the min-max criterion.⁵ Due to the last property, IPW based on the ECB is implemented without further common support restrictions.

Ultimately, choosing an IV estimator involves a trade-off: Standard 2SLS is more easily applied than matching or weighting but 2SLS may be inconsistent under effect heterogeneity. Moreover, recent simulation evidence by Sloczynski et al. (2022) suggests that more flexible estimators may even be competitive in terms of mean squared error compared to standard 2SLS. However, more research on the relative performance of IV estimation methods under realistic data-generating processes is necessary to provide better guidance to researchers.

This Section provides empirical evidence on the relevance of potential inconsistencies in 2SLS estimates when an instrument is only valid conditional on covariates. This is done by re-estimating the wage returns to college exploiting college proximity as instrument using the data originally analyzed by Card (1995).

By re-examining the Card (1995) data on college proximity and the returns to college, this paper shows that potential inconsistencies in 2SLS estimates of local treatment effects documented in the theoretical literature are not merely a hypothetical threat when effects are heterogenous. For the data at hand, 2SLS yields systematically larger effects than more flexible estimators based on the PS with differences amounting to roughly 50 to 100%. It is shown that this is because standard linear-in-covariates 2SLS yields a conditional variance-weighted average effect, putting more weight on units with a PS close to a coin flip. In line with theoretical predictions by Sloczynski (2021), the results suggest that 2SLS estimates can be expected to be more trustworthy when sample shares of instrument groups are roughly of equal size. Moreover, the paper shows that this is because the effects of conditional-variance weighting tend to be less severe when groups sizes are similar. Overall, the results show that the presumption that 2SLS yields point estimates close to more flexible estimators based on the PS as argued by Angrist and Pischke (2008) does not apply in general and that one should be suspicious of 2SLS estimates when group sizes differ substantially and covariates are predictive of the instrument. In that case it may be best to use semi- or non-parametric estimation techniques instead. At the very least, one should use these methods to assess the sensitivity of estimates regarding implicit parametric assumptions made when using linear-in-covariates 2SLS.