Top

Empirical Economics

Published in:

14-05-2020

Does the choice of balance-measure matter under genetic matching?

Authors: Adeola Oyenubi, Martin Wittenberg

Published in: Empirical Economics | Issue 1/2021

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In applied studies, the influence of balance measures on the performance of matching estimators is often taken for granted. This paper considers the performance of different balance measures that have been used in the literature when balance is being optimized. We also propose the use of the entropy measure in assessing balance. To examine the effect of balance measures, we conduct a series of simulation studies where we optimize balance using genetic algorithm (GenMatch). We found that balance measures do influence matching estimates under the GenMatch algorithm. The bias and root-mean-square error (RMSE) of the estimated treatment effect vary with the choice of balance measure. While the performance of different balance measures vary across simulation designs, some pattern did emerge. In an artificial data generating process (DGP) with one covariate, the proposed entropy balance measure has the lowest RMSE. However, in more realistic DGPs with many covariates, the standardized difference in means appear to be a very robust measure of balance, this measure either dominate other measures or come in as a close second in terms of bias or/and RMSE. The implication of these results is that sensitivity of matching estimates to the choice of balance measure should be given greater attention in empirical studies.

previous article Optimal pricing of alpine ski passes in the case of crowdedness and reduced skiing capacity

next article Intergenerational transmission of risk attitudes in Burkina Faso

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

The traditional implementation of propensity score matching requires discussion of the propensity score specification (see Lehrer and Kordas 2013 and their discussion on estimation of propensity scores). This, in itself, may affect the results, apart from the impact of balance measures. Here we fix the matching method and check if the result is sensitive to the choice of balance measure.

This balance measure—the entropic distance metric between distributions, was originally proposed by Oyenubi (2018) in his PhD thesis, under the supervision of Prof. Martin Wittenberg and Prof. Patrizio Piraino.

This number is called population size in Genetic Algorithm.

This fitness function can be to minimize the mean of the balance statistics across all covariates or perform lexical optimization (see Diamond and Sekhon 2013).

Selection gives preference to good solutions to make it into the next generation of solutions (or the offspring population). Crossover combines two or more current solutions to form a new solution (offspring in the new population). Mutation is used to encourage diversity amongst solutions. This is achieved by changing parts of a candidate solution in the current population randomly to produce new solutions. See Mebane and Sekhon (2011) for more details.

We do not expect this extreme case in our context because of the common support assumption. However, there may be areas of thin/no support in finite samples which this measure will be useful in picking up.

The (average) treatment to control ratio for Simulation 1 is displayed in online Appendix B. For Simulation 2, the probability of treatment assignment at the average value of the covariates is approximately 0.5.

i.e., \(\% {\text{Bias}} = \left| {\frac{{\mathop \sum \nolimits_{i = 1}^{500} \frac{{\left( {\hat{\theta } - \theta } \right)}}{500}}}{\theta }} \right|* 100\) where \(\theta\) is the true ATT and \(\hat{\theta }\) is the estimate of \(\theta\) in each iteration of the simulations. The RMSE is given by \(\mathop \sum \nolimits_{i = 1}^{500} \frac{{\left( {\hat{\theta } - \theta } \right)^{2} }}{500}\). For simulation 1 \(\theta\) is replaced with the absolute value of the minimum Bias estimate.

Note that for simulations 2 and 3 we use a subset of the balance measures used in simulation 1. Since Simulations 2 and 3 has discrete and continuous covariates only balance measures that can handle both kinds of variables can be used.

This is because Mean and Variance are sufficient to identify normal distributions. Measures that look beyond these two quantities will, therefore, be inefficient.

The only exception is the default balance measure in simulation 3 under bias correction and φ = − 1.5.

Austin PC (2009) Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med 28(25):3083–3107CrossRef

Belitser SV, Martens EP, Pestman WR, Groenwold RH, Boer A, Klungel OH (2011) Measuring balance and model selection in propensity score methods. Pharmacoepidemiol Drug Saf 20(11):1115–1129CrossRef

Busso M, DiNardo J, McCrary J (2014) New evidence on the finite sample properties of propensity score reweighting and matching estimators. Rev Econ Stat 96(5):885–897CrossRef

Caliendo M, Kopeinig S (2008) Some practical guidance for the implementation of propensity score matching. J Econ Surv 22(1):31–72CrossRef

Carr J (2014) An introduction to genetic algorithms. Sr Proj 1:40

Casella G, Berger RL (2002) Statistical inference, vol 2. Duxbury, Pacific Grove

Crump RK, Hotz VJ, Imbens GW, Mitnik OA (2009) Dealing with limited overlap in estimation of average treatment effects. Biometrika 96(1):187–199CrossRef

Dehejia RH, Wahba S (1999) Causal effects in nonexperimental studies: reevaluating the evaluation of training programs. J Am Stat Assoc 94(448):1053–1062CrossRef

Diamond A, Sekhon JS (2013) Genetic matching for estimating causal effects: a general multivariate matching method for achieving balance in observational studies. Rev Econ Stat 95(3):932–945CrossRef

Frölich M (2004) Finite-sample properties of propensity-score matching and weighting estimators. Rev Econ Stat 86(1):77–90CrossRef

Granger C, Maasoumi E, Racine J (2004) A dependence metric for possibly nonlinear processes. J Time Ser Anal 25(5):649–669CrossRef

Hainmueller J (2012) Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Polit Anal 20(1):25–46CrossRef

Ho DE, Imai K, King G, Stuart EA (2007) Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Polit Anal 15:199–236CrossRef

Hoaglin DC, Mosteller F, Tukey JW (1983) Understanding robust and exploratory data analysis, vol 3. Wiley, New York

Huber M (2009) Testing for covariate balance using nonparametric quantile regression and resampling methods. Unpublished working and discussion papers

Huber M, Lechner M, Wunsch C (2013) The performance of estimators based on the propensity score. J Econom 175(1):1–21CrossRef

Iacus SM, King G, Porro G, Katz JN (2012) Causal inference without balance checking: coarsened exact matching. Polit Anal 20(1):1–24CrossRef

Imai K, Ratkovic M (2014) Covariate balancing propensity score. J R Stat Soc Ser B (Stat Methodol) 76(1):243–263CrossRef

Imai K, King G, Stuart EA (2008) Misunderstandings between experimentalists and observationalists about causal inference. J R Stat Soc Ser A (Stat Soc) 171(2):481–502CrossRef

Khan S, Tamer E (2010) Irregular identification, support conditions, and inverse weight estimation. Econometrica 78(6):2021–2042CrossRef

King G, Lucas C, Nielsen RA (2017) The balance-sample size frontier in matching methods for causal inference. Am J Polit Sci 61(2):473–489CrossRef

Kinnear Jr KE (1994) A perspective on the work in this book. In: Advances in genetic programming, p 3–19

Kvam PH, Vidakovic B (2007) Nonparametric statistics with applications to science and engineering, vol 653. Wiley, New YorkCrossRef

LaLonde RJ (1986) Evaluating the econometric evaluations of training programs with experimental data. Am Econ Rev 76(4) :604–620

Lechner M, Strittmatter A (2019) Practical procedures to deal with common support problems in matching estimation. Econom Rev 38(2):1–15

Lee BK, Lessler J, Stuart EA (2010) Improving propensity score weighting using machine learning. Stat Med 29(3):337–346CrossRef

Lehrer SF, Kordas G (2013) Matching using semiparametric propensity scores. Empir Econ 44(1):13–45CrossRef

Maasoumi E, Racine JS (2008) A robust entropy-based test of asymmetry for discrete and continuous processes. Econom Rev 28(1–3):246–261CrossRef

Maasoumi E, Wang L (2019) The gender gap between earnings distributions. J Polit Econ 127(5):2438–2504CrossRef

Mebane WR, Sekhon JS (2011) Genetic optimization using derivatives: the rgenoud package for R. J Stat Softw. https://doi.org/10.18637/jss.v042.i11CrossRef

Mitchell M (1998) An introduction to genetic algorithms. MIT Press, CambridgeCrossRef

Oyenubi A (2018) Quantifying balance for causal inference: an information-theoretic perspective. Doctoral dissertation, University of Cape Town

Parizzi A, Brcic R (2011) Adaptive InSAR stack multilooking exploiting amplitude statistics: a comparison between different techniques and practical results. IEEE Geosci Remote Sens Lett 8(3):441–445CrossRef

Sekhon, J. S., 2011. Multivariate and propensity score matching software with automated balance optimization: The matching package for R

Setoguchi S, Schneeweiss S, Brookhart MA, Glynn RJ, Cook EF (2008) Evaluating uses of data mining techniques in propensity score estimation: a simulation study. Pharmacoepidemiol Drug Saf 17(6):546–555CrossRef

Tan YP, Nagamani J, Lu H (2003) Modified Kolmogorov–Smirnov metric for shot boundary detection. Electron Lett 39(18):1313–1315CrossRef

Zhao Z (2004) Using matching to estimate treatment effects: data requirements, matching metrics, and Monte Carlo evidence. Rev Econ Stat 86(1):91–107CrossRef

Title: Does the choice of balance-measure matter under genetic matching?
Authors: Adeola Oyenubi
Martin Wittenberg
Publication date: 14-05-2020
Publisher: Springer Berlin Heidelberg
Published in: Empirical Economics / Issue 1/2021
Print ISSN: 0377-7332
Electronic ISSN: 1435-8921
DOI: https://doi.org/10.1007/s00181-020-01873-9

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 1/2021

The robustness of forecast combination in unstable environments: a Monte Carlo study of advanced algorithms

Optimal pricing of alpine ski passes in the case of crowdedness and reduced skiing capacity

Financing constraints and exports: Evidence from manufacturing firms in India

The impact of fiscal consolidations on growth in sub-Saharan Africa

Higher frequency hedonic property price indices: a state-space approach

Exchange rate pass-through to import prices in Europe: a panel cointegration approach