What we measure when we measure the effects of user fees: a replication, reanalysis, and extension of Tanaka, 2014

In the literature on the effects of charging user fees for healthcare on health outcomes, Shinsuke Tanaka’s 2014 AEJ: Economic Policy paper evaluating South Africa’s 1994 user fee reform, “Does Abolishing User Fees Lead to Improved Health Status? Evidence from Post-Apartheid South Africa,” is regarded as a seminal piece of evidence demonstrating that eliminating user fees significantly improves population health. Tanaka finds that South Africa’s elimination of user fees for pregnant women and young children caused children’s weight-for-age z-scores (WAZ) to rise by 0.6, making it one of the most successful child nutrition interventions ever studied. However, evaluations of the same reform employing other methodologies have found that it had little to no effect on outcomes like healthcare utilization, making it difficult to understand how it might have had such large effects on WAZ. In this paper, I replicate, reanalyze, and extend Tanaka, 2014, considering its use of a two-way fixed effects model (TWFE) with additional covariates in the context of recent methods literature exposing the limitations of such designs. I show that Tanaka’s approach may not yield an unbiased causal estimate under the intended conditional parallel trends assumption but find that his results are robust to reanalysis with updated estimators. I then explore the implications of substantial treatment effect heterogeneity across subgroups within Tanaka’s sample for the external validity of his results, arguing that this heterogeneity severely undermines the case that his results generalize. I thereby recontextualize the debate over the effects of eliminating user fees in South Africa in terms of new thinking about TWFE and treatment effect heterogeneity, illustrating the practical implications of those advances in the process.