When do performance assessments influence policy behavior? Micro-evidence from the 2014 Reform Efforts Survey

Among other things, the Saakashvili administration fired 15,000 members of the notoriously corrupt police force, dramatically increased the salaries of 10,000 public servants to combat petty corruption, reduced the time and cost of property registration by 75%, cut the corporate income tax rate from 20% to 15%, streamlined customs by reducing the number of days it takes to export from 54 to 10, and slashed the number of activities requiring a business license from 909 to 159 (Bagaudinova et al. 2007; World Bank 2012; Bendukidze and Saakhashvili 2014; Light 2014). These and other reforms resulted in major improvements on the World Bank’s Doing Business Indicators, and USAID (2009) celebrated Georgia’s rapid rise on the Doing Business indicators as evidence of “the broadest, deepest, fastest business climate reforms [undertaken by] any country in the last 50 years.”

Several studies indicate that the Doing Business Report directly inspired and guided the design and implementation of business environment reforms by the Saakashvili administration (IEG 2008: 46; IEG 2009: 36; USAID 2009; Schueth 2011: 64). Evidence from the 2014 Reform Efforts Survey also suggests that the Doing Business Report exerted outsized influence on the policy priorities and plans of Georgian officials during the Saakashvili administration (see Figure I-2).

During the same period of time, the Saakashvilli administration participated in various IMF- and World Bank-monitored reform programs. It signaled its interest in joining the EU and deepened its ties with EU by agreeing to a European Neighborhood Policy Action Plan in 2006 and an Association Agreement in 2010 (Di Puppo 2014). It also made clear that it hoped to join NATO and was willing to adjust its policies to comply with NATO’s expectations. Many rationalist and constructivist IR scholars would interpret such actions as efforts to signal commitment and credibility to external actors and thereby unlock material and reputational rewards (Chwieroth 2013; Girod and Tobin 2016; Buntaine et al. 2017). Yet curiously, the Georgian government did not choose to emphasize any of these efforts in their publicity campaign. They instead focused almost exclusively on the country’s performance vis-à-vis its competitors and peers, as measured by World Bank’s Doing Business Report. This strategic choice calls attention to two closely-related puzzles that we try to solve in this article: why are cross-country PAs increasingly popular credibility signaling devices, and when and why do countries use cross-country PAs to guide their reform priorities and plans?

Here we use the term “performance assessments” to refer to discrete, codified, and regularized assessments of government performance that are conducted by one or more third-party entities in one or more policy domains. We only consider PAs that are national in scope. We do not consider assessments that evaluate specific programs, projects, or subnational entities.

These policy outcomes include, among other things, the passage of anti-trafficking and anti-money laundering legislation, increased rates of primary school enrollment, lower levels of public sector corruption, and higher levels of state compliance with international labor standards.

Elaborating on this point, Grindle and Thomas (1991: 68) explain that “[one] needs to be centrally concerned with the attitudes, perceptions, and actions of these elites, since they [are] fundamental to any efforts to initiate and sustain reform.”

To date, this literature has largely focused on establishing the existence of causal relationships between cross-country PAs and policy outcomes of interest. This initial focus on whether PAs influence final policy outcomes is understandable a first-order question is whether PAs are epiphenomenal or inconsequential. However, now that we know “assessments matter,” it has become increasingly clear that we need to need to better understand when these assessments matter and the causal processes through which they operate.

States, of course, also seek to send credibility signals to foreign donors (Dreher et al. 2012; Knack 2013; Buntaine et al. 2017). However, in allocating the money and external validation that recipient countries value, donors respond primarily to factors other than recipient performance (Alesina and Dollar 2000; Easterly and Williamson 2011). Therefore, incentives to send signals of credibility, or good performance, may be weaker for donors than for investors.

There is both observational and experimental evidence that foreign investors use the Doing Business Report in this way. Martin et al. (2014) find that investor sentiment, as measured by local stock market indices, improves in response to improvements on the World Bank’s Doing Business indicators. Corcoran and Gillanders (2015) provide evidence that improvements on the World Bank’s Doing Business indicators result in higher FDI inflows. Kelley et al. (2017) conducted an experiment with 150 professional investors and found that exposure to information about country rankings on World Bank’s Doing Business indicators substantially affected the willingness of investors to make investments in emerging markets. There is also evidence that cross-country PAs other than the Doing Business Report have similar information-updating impacts among donors, creditors, and investors (Öhler et al. 2012; Morse 2017).

According to Acemoglu et al. (2013), “[r]eform programs in numerous countries use the Doing Business measures as an input into policymaking. Over 350 regulatory reforms across the [w]orld have been inspired by Doing Business since 2003. Some 35 governments currently maintain inter-ministerial committees considering reforms informed by Doing Business. More broadly every year many governments have undertaken reforms leading to improvements of the indicators without explicitly considering the Doing Business report. The total of reforms tracked by Doing Business annually exceeds 200 globally—over 2000 since the report tracks them. This suggests that between 15 and 20% of all such reforms were affected by the report itself” (Acemoglu et al. 2013).

There are also journalistic accounts that suggest foreign investors pay attention to the Doing Business Report and consider it to be a meaningful signal. For example, in 2012, the New York Times published a story about an investment fund manager who decided to double down on investments in Egypt after monitoring the country’s performance on the World Bank’s EDBI. He said that “[o]ne of the things that attracted us to Egypt was that the World Bank had improved its ‘ease of doing business’ rating. ... From 2004 to 2007, it had the single greatest improvement in ‘ease of doing business’ of any country in the world” (Luongo 2012).

The fact that many policymakers perceive cross-country PAs as imposing unreasonably high domestic policy autonomy costs is clearly evident in the debates that have surrounded the World Bank’s Doing Business Report. In 2013, the World Bank commissioned an independent panel review of this particular cross-country PA. A number of the developing countries, including China, used the opportunity of the independent panel review to advocate for the elimination of the Doing Business Report (Harding 2013). One interview that the independent panel conducted with a senior official from a World Bank borrower country nicely summarizes the “one-size-fits-all” criticism, which motivated the effort to abolish the PA: “The report is fundamentally biased. It assumes that deregulation is the more appropriate approach to follow and limited liability companies are the more appropriate business form to take. It also expects meaningful changes every year regardless of whether these changes fit into the long-term and more consistent reform program and regulatory framework of countries. Yet the regulatory function of the procedures and the standards of implementation and enforcement of regulations, which are equally if not more important, are not taken into consideration …” (World Bank 2013).

Of course, while participatory PAs have knowledge and legitimacy advantages, it is also possible that involving assessed entities in assessment processes could lead to capture, manipulation, and status quo bias (Rodrik 1996; Swedlund 2013). However, our study is focused on explaining the determinants of assessment influence. On the broader issue of whether external assessments of government performance lead to negative, unintended consequences, see Parks et al. (2015).

Indeed, the evidence uptake and organizational learning literatures suggests that the most influential diagnostic and advisory tools are those that involve “the assessed” in evaluation design, data collection, and evidence interpretation processes (Patton 2008; Russ-Eft and Preskill 2009).

In Section 5 of this study, we analyze attributional responses from policymakers who have firsthand knowledge of the reasons why their governments recalibrated reform priorities in response to cross-country PAs and country-specific PAs. We find that respondent evaluations of the “reasons for assessment influence” in the cross-country benchmarking cohort are statistically less likely than the comparison group (respondent evaluations of PAs that did not involve cross-country benchmarking) to indicate that these assessments were “seen as respecting the [country’s] sovereign authority over final policy decisions.” A likely explanation for this empirical pattern is that government officials perceive cross-country benchmarking exercises as promoting one-size-fits-all policy solutions (Arruñada 2007; Chhotray and Hulme 2009; Goldsmith 2011; World Bank 2013).

An extensive literature in international relations demonstrates that bad reputations and good reputations are economically and politically consequential (e.g., Crescenzi 2007). For example, if a state has a reputation for default on it loans, it will be significantly more difficulty accessing international capital markets and securing new loans from external sources on favorable terms (Tomz 2007).

Reform is more likely when it becomes “the business of a wider range of insiders” (Jacoby 2006: 638) and when the public places pressure on the authorities to follow-through on their reform commitments (Abiad and Mody 2005).

These survey data are valuable in that they capture the experiences and observations of public sector officials from the governments that assessment producers ultimately wish to influence. While we cannot be certain that all survey respondents provided answers on the basis of policy decisions that they themselves made or directly witnessed, the types of respondents who participated in the survey are substantially more proximate to the empirical phenomena of interest (PA influence on policy decisions) than any other population. A public opinion survey or a survey of experts would have been significantly less useful, as these populations generally know little about the inner-workings of government (Hoffmann-Lange 2007; Tansey 2007).

See Online Appendix A for more information on how the sampling frame was constructed for this survey and how the survey was conducted.

Most of these studies explicitly account for self-selection bias—i.e., the fact the policymakers might use PAs to validate reforms that they were going to undertake anyway.

In Figure I-1, we report the average levels of reported agenda-setting influence (across our full sample of 123 countries) for the performance assessments in our sample.

See Parks et al. (2015) for more details on the design and content of the 2014 Reform Efforts Survey.

Readers might question whether these survey-based measures of PA influence capture any meaningful information about the actual implementation of de facto reforms. Indeed, since individual respondents evaluated PA influence based on their own personal observations and experiences, it is possible that these evaluations do a poor job of capturing the true scope of influence that a specific PA exerted on a specific reform outcome or set of reform outcomes. Therefore, to test the construct validity of dependent variable, we examined whether our survey-based measure of perceived PA influence strongly correlates with objective measures of de facto reform implementation. To do so, we selected a PA that was evaluated by a large number of our survey respondents (across many countries and years) and that itself produces objective measures of reform implementation for most of the same countries and years: the World Bank’s Doing Business Report. If it is true that our dependent variable measures what it purports to measure, one would expect respondent evaluations of the Doing Business Report’s policy influence to correlated with objective measures of business environment reform implementation from the Doing Business Index. In Online Appendix G, we confirm that respondent evaluations of the policy influence of the Doing Business Report are strongly and positively correlated with the actual implementation of business environment reforms as reflected in the actual changes in Doing Business Index. Parks and Davis (forthcoming) conduct a similar construct validity test for a survey-based measure of the policy influence of the Millennium Challenge Corporation’s eligibility criteria and country scorecards (another PA included from our sample). They too report a strong, positive correlation between respondent evaluations of PA influence and independently-generated measures of reform adoption and implementation.

The fact that the sample for this study consists of 1784 government officials rather than the 3400 government officials who participated in the survey is mostly a reflection of the fact that not all government officials were knowledgeable about individual PAs. The 2014 Reform Efforts Survey was designed to only allow those government officials who were knowledgeable about individual PAs to answer questions about their perceived level of agenda-setting influence (in question 31) and reform design influence (in question 32). Tables A-1, A-2 and A-3 in Online Appendix A provide summary statistics on the characteristics of the 1784 government officials who participated in the 2014 Reform Efforts Survey and assessed at least one of the individual assessments.

Survey participants were first asked (in question 29) to identify a list of PAs with which they were familiar. They selected these assessments from a fixed list (see Parks et al. 2015 and Online Appendix C for more details). They were subsequently asked (in question 30) to write-in any additional PAs that they believed might have informed the government’s reform efforts (in the respondent’s particular area of policy expertise. They then individually evaluated (in question 31) the agenda-setting influence of each of these assessments on a scale of 0–5.

In Online Appendix H, we report the average agenda-setting influence score of each of the 103 assessments as evaluated by public officials who took part in the 2014 Reform Efforts Survey.

Readers will recall from section II of this article that we have no theoretical basis to expect the credibility signaling benefits of cross-country benchmarking exercises to be larger—or smaller—than their policy autonomy costs. Without detailed information about the size of the benefits of credibility signaling and the costs of foregone policy autonomy, one cannot know with any certainty whether benefits exceed costs (and the cross-country PA will exert influence) or costs exceed benefits (and the cross-country PA will fail to exert influence). Therefore, the direction of the relationship between cross-country benchmarking and policy influence is itself an empirical question to be tested.

In cases when the assessed government can increase its policy autonomy (i.e., higher sovereignty and lower domestic audience costs) and reduce its uncertainty about the achievability of the net benefits (of signaling credibility to investors and donors), we expect risk-adjusted cost-benefit calculations to shift in the direction of taking policy action to address the performance requirements of a cross-country PA. This also means that when a cross-country benchmarking exercise allows for assessed governments to engage in the assessment process, it is effectively reducing the risk adjustment factor that assessed governments used in their cost-benefit calculation (of whether or not to address the performance requirements of the assessment). This risk adjustment factor, in effect, represents the imperfect information that the assessed government possesses about the likelihood of reaping the credibility signaling benefits that cross-country assessments can provide.

Some of the PAs that focus exclusively on inputs include Extractive Industries Transparency Initiative (EITI) requirements, the membership requirements of the Egmont Group of Financial Intelligence Units, and the World Trade Organization’s Accession Working Party Reports and Accession Protocols.

Roughly 49% of the variation in our dependent variable can be “explained” with respondent-fixed effects, which suggests that it is important that we account for respondent-level attributes.

All of the regressions also control for respondents’ policy areas of focus and the types of positions that they held. However, for the sake of brevity, the regression coefficients of these respondent-level variables are not reported in our regression tables. Each respondent was asked to select his or her primary policy area of focus (from a fixed list 23 different policy areas) and specify a position that he or she held during their period of public service in a given country between 2004 and 2013 (which were subsequently recoded into a standardized set of “position types”; see Table A-2 in the Online Appendix). Respondents were then asked to evaluate each PA from the vantage point of his or her policy area of focus and country in that particular position and time period (Parks et al. 2015).

Our models also reveal several other interesting results. Consistent with previous research, we find that smaller countries, poorer countries, and more aid-dependent countries are more responsive to external sources of data and analysis (Andrews 2011; Parks et al. 2015; Custer et al. 2016; Parks and Davis forthcoming; Buntaine et al. 2017). We also find some evidence for the claim that domestic political support for reform conditions assessment influence (Chassy 2014). More specifically, we find that reform support by the Office of the Head of State and/or Office of the Head of Government (EXECUTIVE SUPPORT) has a statistically significant and positive effect on assessment influence (p value<0.01). These findings attest to the centrality of the executive branch in leading and shepherding (PA-informed) reform processes. We also find that presence of reform supporters within the judicial branch (JUDICIAL SUPPORT) and domestic civil society organizations (CIVIL SUPPORT) can expand or narrow the scope of influence that performance assessments exert on reform priorities and processes. These results may point to “pass-through effect” whereby a country’s propensity to undertake reform amplifies or dampens the policy influence of PAs (Chassy 2014).

These tests are described at greater length in Online Appendix F.

Participants in this survey were only asked to answer this question for PAs that received the highest sum of ratings in question 31 (agenda-setting influence) and 32 (design influence). When they tied in the sum, respondents were asked to answer this question only for the assessment that received a higher rating in question 31. If they still tied in question 31, one of the assessments was selected at random.

These controls are included to account for characteristics of respondents or the countries where respondents live and work that might systematically bias the ways in which individuals characterize the reasons for PA influence (e.g., social desirability bias).

Survey responses from those with firsthand knowledge of the reasons why their governments recalibrated reform priorities in response to cross-country PAs suggest that policymakers weigh the benefits of credibility signaling to investors against the costs of foregone policy autonomy. However, it is important to note that there is no evidence that these same considerations influence the decision-making of governing elites in developing countries when they evaluate if it is in their best interests to address the performance requirements of a country-specific PA.

Earlier in the survey, respondents were informed that the term “development partners” referred to “international organizations, foreign embassies, and development finance agencies.”

The primary factors that shape the cross-country allocation of aid are donor interests and recipient needs (Alesina and Dollar 2000; Easterly and Williamson 2011).

In cases when donors do allocate aid on the basis of recipient performance, they often link their financial disbursements to performance on country-specific assessments rather than cross-country benchmarking assessments (Savedoff 2011).

Indeed, none of the country-specific assessments in our sample (see Online Appendix C) are produced by foreign investors.

See also Table G-1 in the Online Appendix.

Kakha Bendukidze, the country’s State Minister of Reform Coordination, seized upon the cross-country rankings in the World Bank’s annually-generated Doing Business Report as a uniquely useful means by which the government could signal its credibility to foreign investors (Schueth 2011; De Waal 2011).

They did so with support from USAID’s Georgia Business Climate Reform (GBCR) project (USAID 2009). As explained by Schueth (2011: 64), “GBCR was the preferred partner of [Minister] Bendukidze and [the Office of the State Minister on Reforms Coordination] because its mandate enabled the flexible allocation of resources in support of the government’s changing priorities. By June 2006, continuous consultation and collaboration between [the Office of the State Minister on Reforms Coordination] and GBCR led to the convergence of both organizations’ priorities on making sure that EDBI reflected the progress already being made in regulatory reform. GBCR replicated EDBI’s methodology using data that it collected independently using business process mapping methods—e.g., GBCR staff counted the number of documents and days necessary to clear goods through Georgian customs terminals, and visited the offices, filled the forms, and paid the fees necessary to register a new business. GBCR compared the results of its independent research with the indicators published for Georgia in the 2006 edition of EDBI.”

According to a 2008 report from the “Doing Business [DB] creates its annual country rankings using information supplied by persons deemed knowledgeable about selected laws and regulations in each country covered. The DB team identifies individual lawyers, notaries, officials, and firms and requests that they provide information on one or more specified DB topics” (IEG 2008: 13). The same report notes that approximately 18% of the contributors to the Doing Business Report are government officials from assessed countries.

Some argue that these material consequences motivate policy responses by states (Noland 1997; Öhler et al. 2012). Others argue that rather than making a rational calculation of whether a particular type of policy adjustment serves their own interests or the national interest, public sector decision-makers take “cognitive shortcuts” and look to their neighbors or other peers for information about the most “modern” and “appropriate” types of policies (Weyland 2005; Gray 2013).

In fact, the collection and publication of information about state performance can lead to “gaming” or other forms of opportunistic behavior (Sandefur and Glassman 2015; Kerner et al. 2017; Buntaine et al. 2017).