Micro-econometric and Micro-Macro Linked Models: Impact of the National Agricultural Advisory Services (NAADS) Program of Uganda—Considering Different Levels of Likely Contamination with the Treatment

The data are from two rounds of household surveys conducted in 2004 and 2007. The 2004 survey served as the baseline on which a stratified sample was based according to the year when the NAADS program was first implemented in the community (sub-county) where the program: began in 2001/02; began in 2002/03; began between 2005 and 2007; or had never been implemented at the time of the 2007 survey. This was done to account for the effect of the rollout of the program that may result in a modified treatment among later entrants to the program due to learning from previous treatments among earlier entrants of the program (supply-side effects of the program), as well as from nonrandom preparedness of later entrants prior to receiving the treatment (demand-side effects). About 400, 300, 100, and 100 households were surveyed from each of the four strata (see Benin et al. 2011 for details). This paper is based on the panel of 719 household observations. The indicator of interest for estimating the treatment effect is agricultural income (INC)—details of this and other variables used are presented later.

What we are interested in is the average treatment effect of the treated (ATT _j ):where INC _1j is agricultural income of farm household j due to participation in the program and INC _0j is agricultural income of the same farm household j if it did not participate in it. Although, we cannot observe the counterfactual, the underlying estimation problem can be represented as a treatment-effects model of the form: where: \( {NAADS}_j^{\ast } \) is a latent unobserved variable whose counterpart, NAADS _j , is observed in dichotomous form; x _j and w _j are vectors of variables determining agricultural income and the decision to participate in the program, respectively; NAADS _j  = 1 and NAADS _j  = 0 represent participation (or treatment) and non-participation (or control), respectively; α and τ capture the individual and time specific effect, respectively; β and γ are the vectors of parameters measuring the relationships between the dependent and independent variables; ϵ and u are the random components of the equations with joint normal distribution of means (μ, 0) and covariance matrix \( \left[\begin{array}{cc}\hfill {\sigma}_{\varepsilon}^2\hfill & \hfill {\sigma}_{\upepsilon u}\hfill \\ {}\hfill {\sigma}_{\upepsilon u}\hfill & \hfill 1\hfill \end{array}\right] \).

We apply a two-stage weighted regression (2SWR) method (e.g. Robins and Rotnitzky 1995). In the first stage, we estimate Eq. (3) by probit to obtain propensity scores, which are used in selecting a matched sub-sample of treatment and control observations. In the second stage, the propensity scores are used as weights in a weighted least squares regression of Eq. (2) on the matched sub-sample according to: where: ΔINC = INC _t1 – INC _t0, and INC _t0 and INC _t1 are the incomes in the initial (2004) and later (2007) periods, respectively; Δx = x _t1 – x _t0, and x _t0 and x _t1 are the initial and later period values of the covariates, respectively. Equations (5) and (6) represent specifications without and with the covariates, and the impact of the program is measured by \( {\widehat{\delta}}_B \) and \( {\widehat{\delta}}_F \) for the two model specifications, respectively. In any two-stage estimation procedure, it is important to address the identification of the second-stage regression or endogeneity of the first-stage regression. A common procedure used is excluding some of the explanatory variables used in estimating the first-stage probit from the second-stage regression (i.e. having x ⊂ w or x ≠ w and corr (w, ε/x) = 0). In general, nonlinearity of the first-stage probit model renders exclusion restrictions unnecessary (Wilde 2000). Further, since we apply a fixed-effect or difference estimator in the second-stage regression, the condition is satisfied in the sense that Δx ≠ w.

Participation is measured using the status observed in 2007 to avoid crossover in different years so that a treatment household is always a treatment household and cannot switch status; the same for a control household. Of the 719 observations, 66 are treated and 653 are controls, which we split into three. The first control sub-group is made up of those in the same area where the program is implemented and claimed to have benefited indirectly from the program, labeled NAADS_NON-1. The second sub-group also is made up those in the area where the program is implemented but did not claim any benefits (labeled NAADS_NON-2), while the third sub-group is made up those in areas where the program was never implemented (labeled NAADS_NON-3). These three sub-groups make up 256, 284, and 113 observations, respectively. Because matching with the nearest neighbor only can limit any potential gain from matching participants with more than one non-participant with similar attributes, we consider and report estimated treatment effects based on matching with one, three, and five nearest neighbors.

Selected results on common support and balancing tests for different combinations and transformations (squared and interaction terms) of the covariates using matching with three nearest neighbors are shown in Fig. 1 and Table 1, respectively. We find that different covariates and transformations yield different outcomes of common support and balance between the two groups after matching. The general pattern is a skewness of the propensity scores toward one for participants and zero for non-participants. The situation is most perverse when no transformations of the variables are included or when the covariates on the length of program presence are included (Fig. 1a–c). Regarding the latter, different propensity scores are generated for different controls who are identical in all aspects except location in a NAADS sub-county and several treated observations have to be dropped to improve common support. The models associated with the probits when we include transformations of the covariates and exclude the covariates on the length of program presence are preferred because their results show that there is greater common support and only up to six treated observations have to be dropped in any sub-sample (Fig. 1d–f). The sample characteristics in Table 1 also show that any statistically significant differences that existed between the treated and control groups prior to the matching were eliminated or reduced. Together, the results suggest that pooling observations for the different unique control groups as done in Benin et al. (2011) could lead to different policy implications and, as we shall see next, limit any potential gain in knowledge from matching each participant with multiple non-participants that are similar in several attributes but different in others.