Two-stage DEA: caveat emptor

Abstract

This paper examines the wide-spread practice where data envelopment analysis (DEA) efficiency estimates are regressed on some environmental variables in a second-stage analysis. In the literature, only two statistical models have been proposed in which second-stage regressions are well-defined and meaningful. In the model considered by Simar and Wilson (J Prod Anal 13:49–78, 2007), truncated regression provides consistent estimation in the second stage, where as in the model proposed by Banker and Natarajan (Oper Res 56: 48–58, 2008a), ordinary least squares (OLS) provides consistent estimation. This paper examines, compares, and contrasts the very different assumptions underlying these two models, and makes clear that second-stage OLS estimation is consistent only under very peculiar and unusual assumptions on the data-generating process that limit its applicability. In addition, we show that in either case, bootstrap methods provide the only feasible means for inference in the second stage. We also comment on ad hoc specifications of second-stage regression equations that ignore the part of the data-generating process that yields data used to obtain the initial DEA estimates.

Appendix: OLS estimation in BN’s second stage

The first stage estimation in BN’s approach provides an estimator $\widehat{\widetilde{\theta}}_i\le1$ of $\widetilde{\theta}_i$ for $i=1, \ldots,\,n$ where i indexes observations. The properties of DEA estimators have been developed by Korostelev et al. (1995a, b), Kneip et al. (1998), Kneip et al. (2008, 2011b), Park et al. (2010) and Simar and Wilson (2011), and depend on assumptions about returns to scale. In particular, if variable returns to scale (VRS) are assumed, then the DEA estimator converges at rate n ^2/(p+q+1), which is slower than the usual parametric rate n ^1/2 for p + q > 3. BN ignore this in the proof (appearing in BN2) of their Proposition 1, and this leads to important errors and false statements.

BN suggest re-writing (17) as

$$ \log\widetilde{\theta}=\log\widehat{\widetilde{\theta}}-\eta $$

(21)

and using the right-hand side of this to replace $\log\widetilde{\theta}$ in (16) to obtain (18). Then the error term $\widetilde{\delta}$ appearing in (18) is equal to δ + η. BN propose estimating (18) by OLS, and claim in their proof of their Proposition 1 that

$$ \sqrt{n}\left(\widehat{\varvec{\beta}}-\varvec{\beta}\right) \xrightarrow{d} N\left({\bf 0},\sigma^2\varvec{Q}^{-1}\right) $$

(22)

where Q = Plim(n ⁻¹ Z′Z).⁸ As shown below, these claims are false.

Recall that η _i  ≥ 0 for all $i=1, \ldots,\,n,$ with i indexing the sample observations. Simar and Wilson (2011) and Kneip et al. (2011a) prove, under mild regularity conditions,

$$ n^\gamma\eta_i \xrightarrow{{\mathcal{L}}} G(\mu_0,\sigma_0^2), $$

(23)

where $G(\cdot)$ is an unknown, non-degenerate distribution with mean μ₀ > 0 and variance σ ²₀  > 0 (both finite and unknown), and γ = 2/(p + q + 1) for the VRS case (or γ = 2/(p + q) for the constant returns to scale (CRS) case). In addition, as shown in Kneip et al. (2008, 2011a, b), the asymptotic covariances between η _i and η _j is asymptotically non-zero for a number of observations $j=1,\ldots,\,n, j\ne i,$ which is of order O(n ^γ). To summarize, as $n\to\infty,$

$$ E(\eta_i)\approx n^{-\gamma}\mu_0, $$

(24)

$$ \hbox{VAR}(\eta_i)\approx n^{-2\gamma}\sigma_0^2, $$

(25)

and

$$ \hbox{COV}(\eta_i,\eta_j)\approx\left\{\begin{array}{ll} n^{-2\gamma}\alpha&\hbox{for }O(n^\gamma) \hbox{observations} j\ne i;\\ 0&\hbox { for the remaining observations}\end{array}\right\} $$

(26)

for some bounded but unknown constant α.

Recall that the error term $\widetilde{\delta}$ in (18), i.e., the equation that BN estimate by OLS, equals δ + η as shown above. Consequently, the properties of η play an important role in determining the properties of the OLS estimator $\widehat{\varvec{\beta}}$ of $\varvec{\beta}.$ Let ${\varvec{\fancyscript{Z}}}$ be an n × (r + 1) matrix with ith row given by $\left[\begin{array}{ll} 1&-\user2{Z}_i\end{array}\right],$ and let ${\varvec{\fancyscript{Y}}=\left[\begin{array}{lll} \log\widehat{\widetilde\theta}_1&\cdots& \log\widehat{\widetilde\theta}_n\end{array}\right]^{\prime}.}$ In addition, let $\varvec{\beta}^*=\left[\begin{array}{ll}\beta_0&\varvec{\beta}^{\prime}\end{array}\right]^{\prime}$ and $\widehat{\varvec{\beta}}^*=\left[\begin{array}{ll}\widehat{\beta}_0&\widehat{\varvec{\beta}}^{\prime}\end{array}\right]^{\prime}.$ Then OLS estimation on (18) yields

$$ \begin{aligned} \widehat{\varvec{\beta}}^* &=\left({\fancyscript{ Z}}^{\prime}\varvec{{\fancyscript{Z}}}\right)^{-1}\varvec{{\fancyscript{Z}}}^{\prime}\varvec{{\fancyscript{Y}}}\\ &=\left(\varvec{{\fancyscript{Z}}}^{\prime}\varvec{{\fancyscript{Z}}}\right)^{-1}\varvec{{\fancyscript{Z}}}^{\prime} \left(\varvec{{\fancyscript{Z}}}\varvec{\beta}^*+\widetilde{\varvec{\delta}}\right) \end{aligned} $$

(27)

where $\widetilde{\varvec{\delta}}= \left[\begin{array}{lll}\widetilde{\delta}_1&\ldots&\widetilde{\delta}_n\end{array}\right]^{\prime}.$ Taking expectations,

$$ \begin{aligned} E\left(\widehat{\varvec{\beta}}^*\mid\varvec{{\fancyscript{Z}}}\right)&= \varvec{\beta}^*+\left(\varvec{{\fancyscript{Z}}}^{\prime}\varvec{{\fancyscript{Z}}}\right)^{-1}\varvec{{\fancyscript{Z}}}^{\prime}E(\widetilde{\varvec{\delta}}\mid\varvec{{\fancyscript{Z}}})\\ &=\varvec{\beta}^*+\left(\varvec{{\fancyscript{Z}}}^{\prime}\varvec{{\fancyscript{Z}}}\right)^{-1}\varvec{{\fancyscript{Z}}}^{\prime}E(\varvec{\delta}\mid\varvec{{\fancyscript{Z}}})+\left(\varvec{{\fancyscript{Z}}}^{\prime}\varvec{{\fancyscript{Z}}}\right)^{-1}\varvec{{\fancyscript{Z}}}^{\prime}E(\varvec{\eta}\mid\varvec{{\fancyscript{Z}}})\\ &=\varvec{\beta}^*+\left(\varvec{{\fancyscript{Z}}}^{\prime}\varvec{{\fancyscript{Z}}}\right)^{-1}\varvec{{\fancyscript{Z}}}^{\prime}E(\varvec{\eta}\mid\varvec{{\fancyscript{Z}}})\\ &\approx\varvec{\beta}^*+n^{-\gamma}c_1 \end{aligned} $$

(28)

as $n\to\infty,$ where c ₁ is a non-zero, bounded constant, due to the result in (24) and since (by BN’s assumptions) ${E(\varvec{\eta}\mid\varvec{\fancyscript{Z}})=E(\varvec{\eta}), E(\varvec{\delta}\mid\varvec{\fancyscript{Z}})=0,}$ and where $\varvec{\delta}=\left[\begin{array}{lll}\delta_1&\ldots&\delta_n\end{array}\right]^{\prime}$ and $\varvec{\eta}=\left[\begin{array}{lll}\delta_1&\ldots&\delta_n\end{array}\right]^{\prime}.$

From the last line in (28) it is clear that as $n\to\infty,$

$$ \sqrt{n}\left(\widehat{\varvec{\beta}}^*-\varvec{\beta}^*\right)\approx O_p\left(n^{\frac{1}{2}-\gamma}\right), $$

(29)

which is rather different from what is claimed in the proof of Proposition 1 of BN (as noted earlier, BN claim that (19) holds). Recall that γ = 2/(p + q + 1) for the VRS case. As shown below, the left-hand side of (29) converges to a non-degenerate random variable with constant variance for p + q ≤ 3, and to a random variable with variance approaching infinity as $n\to\infty$ for p + q > 3. In the CRS case, γ = 2/(p + q), and hence $\sqrt{n}\left(\varvec{\beta}^*-\varvec{\beta}^*\right)$ converges to a non-degenerate random variable with constant variance for p + q ≤ 4, and to a random variable with variance approaching infinity as $n\to\infty$ for p + q > 4. For p + q = 3 in the VRS case and for p + q = 4 in the CRS case, the left-hand side of (29) converges to a random variable with constant variance, but which is not normally distributed as shown below. In their Monte Carlo experiments, BN considered only the case where p = q = 1 with VRS, and consequently did not notice the errors in their proof of their Proposition 1.

Combining the results in (24–26), and using standard central-limit theorem arguments (see Kneip et al. 2011a for mathematical details), we have

$$ \sqrt{n}\left(\widehat{\varvec{\beta}}^*-\varvec{\beta}^*\right) \xrightarrow{{\mathcal{L}}} N\left({\bf 0},\sigma^2\varvec{Q}^{-1}\right)+\sqrt{n}\zeta_n, $$

(30)

where $\zeta_n$ is a random variable such that $\sqrt{n}\zeta_n=o_p(1)$ if γ > 1/2 or $\sqrt{n}\zeta_n=O_p\left(n^{1/2-\gamma}\right)$ otherwise, and σ² = VAR(δ) = VAR(V) + VAR(U) (as in BN).⁹ The result in (30) is very different from (22), which is the result claimed at the end of the proof appearing in BN2 of BN’s Proposition 1. Although the OLS estimator $\widehat{\varvec{\beta}}^*$ of $\varvec{\beta}^*$ is consistent, (22) cannot be used for valid (asymptotic) inference. Moreover, even if γ = 1/2, (30) contains unknown constants and an unknown, bounded random variable. The left-hand side of (30) does not converge to anything that is bounded if γ < 1/2. Bootstrap methods appear to provide the only feasible avenue toward valid inference or hypothesis testing in the second-stage regression.¹⁰

The preceding discussion also illustrates how the numerous restrictive assumptions imposed on the BN model are crucial for consistency of OLS estimation in the second-stage regression. For example, if Z and U—which determines inefficiency—are correlated, then the error terms δ and $\widetilde{\delta}$ must be correlated with Z, in which case OLS estimation in (18) would yield inconsistent estimates. As another example, if V ^M, the bound on the noise process, is not constant, then OLS estimation may be problematic. If $V^M=\overline{V^M}+\zeta,$ where $\overline{V^M}$ is constant and $\zeta$ is random with $E(\zeta)=0,$ then β₀ can be written as $\beta_0=E(V-U)-\overline{V^M},$ but δ would have to be written as $\delta=V-U-E(V-U)-\zeta.$ If $E(\zeta)\ne0,$ then OLS estimation of β₀ will be biased and inconsistent. Worse, regardless of whether $E(\zeta)=0,$ if $\zeta$ is not independent of Z, then OLS estimation in (18) would yield inconsistent estimates of both β₀ and $\varvec{\beta}.$ If the environmental variables are related to the size of firms, and if the error bounds vary with firm size, the Z and $\zeta$ would clearly be correlated; this is likely to be the case in some applications.

Even more troubling is the assumption that V ^M is finite, which implies that the noise term V is symmetrically truncated at −V ^M and V ^M. Suppose, for example, that V∼ N(0, σ ² _V ), and suppose the researcher has a sample of n iid draws $\{V_1,\ldots,\,V_n\}$ from the N(0,σ ² _V ) distribution. Of course, one can easily find the sample maximum, and the maximum value in a normal sample of finite size will certainly be less than infinity. But, it is necessarily difficult, and maybe impossible, to test whether the distribution is truncated at a finite value. In situations in econometrics where truncated regression is used, the truncation typically arises from features of the sampling mechanism (e.g., survey design) or model structure (e.g., in SW, truncation arises from the fact that inefficiency has a one-sided distribution; it would make little sense to assume otherwise). Imposing finite bounds on a two-sided noise process, however, is a far more uncertain prospect.

If V ^M is infinite, then the first-stage estimation using DEA estimators is inconsistent. From (13), it is clear that if V ^M is infinite, then $\widetilde{\phi}(X)$ must be infinite. Re-arranging terms in (15) indicates that $\widetilde{\theta}=Y/\widetilde{\phi}(X)$ for the case of a univariate output considered by BN; hence if V ^M is infinite, then $\widetilde{\theta}$ is undefined, in which case BN’s second-stage regression is an ill-posed problem without meaning.