4. Objectives-Based Evaluation for Accountability and Learning

The standard criteria used in OBE are relevance, effectiveness, efficiency, sustainability, and considerations of development impact. While specified separately, these criteria also have a great deal of connectivity among them. More broadly, assessments of individual projects or aggregate programs are related to considerations of social inclusion, environmental protection, and governance, but at present these development goals are not directly used as evaluative lenses. Ratings for the standard criteria commonly use a four- or six-point scale, with categories that differ by rating institution. As an example, ratings used by the World Bank and Asian Development Bank (ADB) are summarized in Table 4.1 (IED 2016; IEG 2017a).

Relevance assesses the extent to which the financed activity is suited to a country’s development priorities. It asks whether the objectives of the project are valid and whether the activities, outputs, and project design are consistent with the objectives. Relevance considers analyses and lessons on project design, including the financing instrument and modality, as they relate to the objectives. It is important to consider both what is included in the project and what ought to be included. This criterion also looks into how the project takes into account the work of development partners and other organizations. All this means that a project, by the fact that it is approved, does not automatically qualify as being relevant; the evaluator must assess if it is.

Effectiveness measures the extent to which the project’s intended outcomes are achieved, based on a baseline and a target. For a project to be effective, outcomes should have been achieved or be likely to be achieved, and output targets should also have been substantially achieved. It also looks at factors influencing the achievement of the objectives. This is important for assessing the extent to which the outcomes achieved were a contribution of the project’s interventions. Achieved outcomes need to be plausibly attributable to the project or intervention as established through either IE or a theory-based approach and consideration of the counterfactual. If outcome targets were achieved or are likely to be achieved but output targets were not substantially met, the project will not be considered effective. This is because some outcomes at the national level, such as a higher literacy rate, may be achieved by other interventions than the one being evaluated.

Where projects have multiple objectives and outcomes, the evaluation can assign relative weights to the effectiveness of various objectives and outcomes. For example, a project may be implemented in two distinct geographical areas with one having a higher incidence of poverty. If intervention in the poorer area is seen as a higher priority, the poverty outcome in that area might be given more weight.

Efficiency measures how well resources are used to achieve outcomes. It assesses the project’s economic benefits against economic costs using CBA. In assessing efficiency, CBA, in principle, looks at costs and benefits, including unintended and indirect ones, from a societal viewpoint. Project economic-performance indicators—the economic internal rate of return, net present value, and the cost-benefit ratio—are used to determine whether net gains from investing in a project will be enjoyed by the society following project completion.

In many instances, it is instructive to compare alternative approaches to achieving the same results using a least-cost analysis. The cost estimates should be based on the economic costs incurred to implement the project, as well as provisions for the operation and maintenance of the assets over the expected economic life of the project. Project externalities—which are spillover effects and often unintended consequences such as environmental impacts—should be quantified and valued to the extent possible and incorporated into the calculations.

Sustainability here focuses on the more limited aspect (compared to the broad goal of sustainable development) of the likelihood that an activity will be maintained after donor funding has been withdrawn. It considers the major factors influencing the achievement or non-achievement of sustainability of the project’s results. Since an evaluation is typically carried out during the first few years of a project’s operational life, evaluators must make assumptions about the likely sustainability of operational arrangements, many of which are new, and about probable future operations and maintenance (O&M) arrangements. Even within this narrower confine of just one of the evaluation criteria, however, the environmental effects of a project must also be considered such as the effects on natural resource management, pollution, biodiversity, and greenhouse gas emissions.

This emphasis on environmental protection and climate change mitigation or adaptation is a relatively new requirement in evaluating projects. Climate mitigation is crucial, as the cost of adaptation is high. The less done to mitigate climate change, the more severe and expensive are the consequences. Assessments of sustainability should consider political, economic, institutional, technical, social, environmental, and financial risks. The assessment should also consider the adequacy of risk-mitigation measures. Although environmental- and social-risk avoidance may be part of the effectiveness assessment at some institutions, grave social or environmental consequences of failed mitigation may also affect the sustainability assessment.

Development impact is the positive and negative change generated by a development intervention, directly or indirectly, intended or unintended, and it mirrors the degree to which sustainable development is being achieved. It assesses the overall changes attributable to the intervention, the difference made to the beneficiaries, and the number of people affected. Development impacts to which the project contributes tend to be outside the project’s direct control, and their achievement is often not solely attributable to the project outcomes. Typically, they are dependent on other development efforts. The focus of analysis should be on the contribution of project outcomes to the achievement of the project impacts. Moreover, development impacts can also be due to unforeseen events and positive developments in areas that are outside the project scope. Such impacts should not be attributed to the project.

Multilateral development agencies have been using these criteria since the 1990s. In 1991, the Organisation for Economic Co-operation and Development-Development Assistance Committee (OECD-DAC) issued principles for evaluation of development assistance provided to public sector projects. The principles provided general guidance on evaluating projects financed using development assistance, stating that the aim of evaluation was “to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability” (OECD-DAC 1991). Also emphasized was the importance of formulating the objectives to be achieved at the baseline for high-quality OBEs.

Aggregative evaluations also incorporate these criteria. In response to the growing importance of harmonized evaluations, a workshop was held by the Working Group on Aid of the OECD-DAC on country-program evaluation methodology in 1999, and it was agreed that the criteria for evaluating development finance at the country level could draw on these criteria. Since the 1999 workshop, a common core of good practices for country evaluations has evolved, and gradually these common criteria have been adopted and endorsed by the independent evaluation offices of various Multilateral Development Banks (MDBs).

Much of the OBE work is concerned with the performance of individual projects designed and implemented by governments. Applying the criteria discussed earlier, assessments arrive at conclusions on project success or failure in varying degrees. These success rates also vary a great deal across sectors, across countries, and over time.

The main value of delineating success rates (in addition to providing a judgment on the success in using financing) is in deriving lessons for improving performance going forward. Across thousands of projects financed by government and external financial agencies, some common lessons emerge (as seen in various annual evaluation reports from MDBs).

Several aspects of implementation stand out in making a key difference to project success. Adequate allocation of budget for implementation and allowing for adequate spending on operations and maintenance of infrastructure investments are obvious considerations. The importance of pricing and regulatory policy frameworks is clear in the examples of water pricing or energy regulation. Institutional capacities of implementing countries are also important for project success.

Historically, project success in low-income countries has been seen to be lower than in middle-income countries, but this pattern is by no means universal. Furthermore, this gap has narrowed over time, as seen in the annual evaluation reports of Asian Development Bank and the World Bank. East Asia has been a perennial front-runner when it comes to project outcomes, a result especially driven by China’s strong project performance. Historically, infrastructure (energy in particular) has done well, though social sectors, environment, and multi-sectoral investments have strengthened in many settings over time.

The case of a water supply and sanitation project in Sri Lanka illustrates how these criteria are applied (IEG 2017b). There was uneven access to water and sanitation between urban and rural areas. Therefore, a water supply and sanitation program was implemented in 2004 with the objectives of increasing coverage and achieving effective and sustained use of water and sanitation services in rural communities. The project’s aim would correspond directly with what SDG 6 set out in 2015 for clean water and sanitation. The project was approved in 2003 and closed in 2010. The total costs were estimated to be US$69.1 million.

To prepare the project performance assessment report (PPAR), seven visits to project sites were made. In-depth discussions among focus groups were conducted to gather information to assess the project outcomes. The development outcome of the project was rated moderately satisfactory based on an assessment of relevance, efficacy, and efficiency (Table 4.2).

Relevance of the development objective was rated substantial as it was aligned with both the government’s priorities and with the World Bank’s country assistance strategies with respect to expanding water and sanitation services to the rural population. Project design relevance was also rated substantial. The main components of the project design and its results framework were clear and logically linked to the project’s objectives of increasing service coverage and achieving effective and sustained use of water and sanitation services in rural communities.

The outcome of the first objective under efficacy was rated modest. The original target was not achieved. The number of people provided with access to improved water sources under the project increased by 384,100, but that was only 31 percent of the original target and 48.4 percent of the revised target at completion. The report points to two major reasons: decreased funds owing to the tsunami in 2004 and inflation. The target was not revised sufficiently despite the reduction in funds.

The second efficacy objective, which aimed to achieve effective and sustained water use, was assessed using demand-side outcomes: satisfaction, adequacy, reliability, convenience, and time saving, water quality, affordability, and sustainability of services. The achievement of this second objective was rated substantial. Satisfaction was generally high, with 88 percent of the beneficiaries of the completed water subprojects indicating they were satisfied with their access and that water adequacy had improved. A survey indicated that household connections led to sizable time savings.

The household survey at completion indicated that about 46 percent of the subprojects provided continuous water supply and 78 percent of households received piped water every day. The Ministry of Health’s sampling tests showed that only 44 percent of the subprojects provided water of satisfactory quality. The project achieved relatively high sustainability, but there were cases of water resource depletion and issues of frequent repair needs due to poor technical design in the initial phase.

Efficiency was rated substantial. The project provided access to improved water sources to fewer people compared to the original target, with slightly higher costs than estimated at appraisal. But the economic rate of return computed was still relatively high.

Community-based organizations were the main providers of water supply schemes in rural areas and responsible for operation and maintenance of water supply facilities in villages. Therefore, beyond the core criteria, risk to development outcome was rated significant because many community-based organizations faced technical, financial, and organizational challenges in sustaining the project results. Technical challenges included repair of pumps and water contamination. Furthermore, only a few community-based organizations were seen to be financially sustainable. Finally, some of them suffered from a shortage of volunteers, and the institutional arrangement for supporting them was unclear.

Analogous to public sector projects, private sector projects can be evaluated using OBE. Though some OBE criteria are used for both public and private sector projects, there are important differences in their evaluation. Evaluations of private sector projects focus on business performance, which assesses the effect of the project on its financiers, and economic contribution, which assesses its economic effects more broadly.

Private sector project evaluation, while paralleling that of public sector projects, also has distinct features to account for the public support through MDBs to private businesses (see EBRD 2013; MDBs 2018). Since support to private sector projects is part of the broader activities of MDBs, the project assessment works within the framework provided by the MDBs’ overall mission to promote sustainable private sector investment in developing countries while also achieving social outcomes such as inclusive growth and poverty reduction.

Some MDBs have shifted away from methodologies largely driven by business performance. New approaches are based on market benchmarks and apply OBE criteria such as relevance, effectiveness, efficiency, and sustainability to private sector operations. Relevance and effectiveness in this context focus on both development and business outcomes. Efficiency looks at the financial performance (including achievement of business objectives) and economic performance.

An OBE considers a private sector project against its development outcomes, including business performance. The development impact rating is a synthesis of the impact of the project on the country’s economic and social development. Development impacts are evaluated using a with-versus-without-project comparison, that is, considering (1) what happened with the project and (2), counterfactually, what would have happened without it. When a with-versus-without assessment cannot be done, the costs and benefits to the country can be done on a before-versus-after basis.

At the World Bank Group, a private project’s development outcome is measured across four indicators: project business performance; economic sustainability; environmental, social, health, and safety effects; and contribution to private sector development. Each of these measures a distinct aspect of the project’s performance. The development outcome rating is a bottom-line assessment of the project’s results on the ground, and not an “average” of these four indicators. Each of the categories can be rated on a four- or six-point scale indicating the degree of success.

Interestingly, previous empirical analysis has found a strong connection between project development outcomes and an organization’s financial profitability (IEG 2009). For example, in a cohort of projects financed by the International Finance Corporation (IFC), high/high outcomes (high development outcome and high IFC investment return) were achieved in 66 percent of projects (by the number of projects, not volumes), and another 17 percent had low/low outcomes. This is suggestive of the possibility that focusing on development outcomes is also good business (IEG 2009).

It is important also to assess a project’s impact on people other than the investors (or adopt a stakeholder perspective) such as client companies and their customers, employees, government, competitors, and local residents. Examples of economic benefits and costs accruing to them are ease of access to markets and services, greater market efficiency, contribution to government revenues, contribution to poverty alleviation, social or gender equality, and employment generation.

Compliance with specific environmental requirements should be clearly stated in the OBE of private projects. Environmental requirements help enhance environmental management capacity and produce sound development outcomes and can be considered as proxy for acceptable environmental standards. But the effects on the ground should count most in evaluating development outcomes.

When individual project evaluations are added up to the sector level, as in transport or education, or at the thematic level, such as for the environment or urbanization, they bring out the aggregative impact of various projects. These also help uncover the effect of policies and investments that cut across the sector or theme. For example, a new decentralization policy might affect the collective success rates of public sector projects. Aggregative evaluations also often account for the influence of factors outside of the theme and the sector. For instance, an increase in infrastructure investments can affect the collective performance of education or health projects.

Sector evaluation is a useful tool to assess impact that goes beyond individual projects. Various organizations have conducted a range of sector evaluations, for example, in agriculture (IOE 2017; OVE 2015b), finance (OED 2018), and energy (IDEV 2015).

An example is the evaluation of World Bank-funded projects in education (IEG 2006; see also IFC 2014). This evaluation was based on a synthesis of outcomes of projects that focused on education access and learning, complemented by findings on policies and investments across the sector. The evaluation of individual projects conducted over several years was summarized and then used, along with sector-wide findings, to assess impact at the sectoral level.

The sector evaluation was done on the five criteria discussed earlier. The ratings in Table 4.3 comprise three criteria: development outcome (which is a composite rating of relevance, efficiency, and effectiveness), sustainability, and development impact.

Given the rising need to shift focus away from merely providing access to schools and toward improving the quality of education and accelerating learning, outcomes were evaluated by objectives related to access as well as learning and quality of education. Table 4.4 suggests that support for access to primary education in some respects did well, with serious gaps in improved access for girls.

Further, Table 4.5 indicates gaps in improving learning outcomes and improved educational quality.

Another example can be drawn from an evaluation of World Bank support for transport from 1995 to 2005 (IEG 2007). During this period, there were 642 projects with transport components, carrying a total financial commitment of US$32 billion. This was the first evaluation of the transport sector operation of the World Bank, and it provided insights that could not be obtained by evaluating individual projects given the significant diversity within the sector.

This sector-level evaluation also adopted the five criteria as project-level evaluation: outcome, which is a composite rating of relevance, effectiveness, and efficiency; institutional development impact; and sustainability. Within the evaluation period, performance of the projects improved over time across all indicators (Table 4.6). Ratings were lower when large borrowers were excluded because bigger countries in this sample had better institutional capacities.

To gain further insight, sector evaluations can be disaggregated by region. This helps evaluate which region for a particular sector might be lagging and perhaps require further financing and technical assistance. For example, Fig. 4.2 shows that in the transport sector, South Asia is lagging in terms of outcome, while the Europe and Central Asia region and the Middle East and North Africa region may require support in institutional development.

Country-level evaluation looks at the nationwide impacts of projects and associated activities of strategy and policy exchanges in a country. Sometimes the success of individual projects does not translate into better country performance. And despite project-level weaknesses, country-level dialogue and impact on sustainable development can be disproportionately high. The role of country expertise, quality of policy discussions, interactions with other partners, and the role played by knowledge are all factors determining country-level results.

For large lenders such as the World Bank and the IMF, the country-level program considerations are especially important, whereas for smaller bilateral financiers, country-level impact may not be that large. A variety of evaluations of country and global macrocosmic work is carried out at the IMF (see, e.g., IEO of the IMF 2016) and the United Nations (for instance, IEO of the UNDP 2018).

A country evaluation of India done in 2017 (IED 2017) assessed the performance of ADB strategy and programs for India from 2007 to 2015. ADB provided US$22.1 billion in development loans to India over these eight years, mostly for transport, energy, finance, and water and other infrastructure and services.

A full-fledged OBE approach for a country evaluation would assess the strategic objectives set out at the beginning of a country strategy and to what extent the objectives were achieved. India’s strategic directions present a very broad canvas, and ADB’s presence in the financing in terms of size and scope for this large economy is relatively small. Accordingly, the framework for the country evaluation was modest.

Table 4.7 provides the assessment for the main sectors. The transport program was the largest component in ADB’s sovereign portfolio (35 percent) with a value of US$6 billion. Most of the loan went to construction, upgrading, and rehabilitation of rural roads mostly in lagging states in northern and eastern India, which is consistent with the sector strategy to promote connectivity and increase access of rural areas to towns and cities. Development impacts of the road program were rated satisfactory despite some delays that occurred due to limited capacity of executing agencies and contractors, resulting in efficiency being rated less than efficient.

In energy, the second-largest component of ADB’s sovereign portfolio (29 percent), 33 projects were approved with US$4.9 billion in value. Supporting the government’s strategy to develop a strong national grid and expand and optimize transmission and distribution systems, ADB’s support focused heavily on electricity transmission and distribution (81 percent of the energy sector investment). The energy programs also experienced delays for which efficiency was rated less than efficient. Development impacts, however, were rated satisfactory since all completed projects met their objectives of strengthening the transmission system, reducing losses, improving evacuation capacity and system reliability, and increasing access to electricity.

Water and urban infrastructure and services, finance, and public sector management were considered to have achieved satisfactory development impacts. Accordingly, the evaluation considered the overall achievement of these sectors to be successful.

Table 4.8 shows that the report also assessed ADB’s support for three primary strategic agendas: inclusive economic growth, environmentally sustainable growth, and regional cooperation and integration. The report also assessed special priorities: knowledge solutions and capacity development, gender equality, and catalyzing infrastructure investment and public-private partnerships.

Among the three strategic agendas, inclusive growth was most prominent in ADB’s lending portfolio to India, with some 90 percent of projects intended to contribute directly or indirectly to this objective. This was accomplished most notably through the financing that was made available to economically lagging states. In addition, ADB also supported inclusive growth through incorporating inclusive elements into project design (for instance, slum improvement in urban projects) and ensuring the implementation of projects in sectors with high inclusion impact, such as education, water supply, and rural electrification. About half of ADB’s support program was tagged for environmentally sustainable growth, effected mainly through projects in energy and water and urban infrastructure and services.

A similar country evaluation of development assistance to Brazil was done by the World Bank (IEG 2015; see also OVE 2015a). The support aimed at achieving greater equity, sustainability, and competitiveness, underpinned by strong economic management and governance. The evaluation recommended that the World Bank Group value benefits beyond individual interventions and make expected catalytic impact a major criterion in the design of its future strategy in Brazil.

Thematic evaluations can shed light on the overriding question of how development finance affects issues of inclusion, environmental protection, or governance, which form the core of the SDGs. Taken together, evidence emerging from OBE of individual projects or sectors can feed into the major themes that drive sustainable development in the aggregate. MDBs have done thematic evaluation on a number of themes: the environment and climate change (see, e.g., IDEV 2018; OVE 2014), gender and diversity (e.g., IEO of Global Environment Facility 2017; OVE 2018), and urban development (Cities Alliance 2017).

Consider a case where distributional impact is a concern and three projects—water access improvement, road construction, and deforestation activity control—were financed by an MDB. The water project may benefit the poor much more than the non-poor. Roads, on the other hand, may not connect the remote areas where the poor mostly reside, thus limiting the benefits that the poor can gain from them. It is also possible that controlling deforestation activity may on balance help the poor who depend on the sustainability of forest resources for their livelihood. Thus, different projects and investments in different sectors may affect the poor simultaneously but in different ways. Overall distributional impact can thus be assessed through a thematic evaluation.

Together, the various strands of OBE can be brought to bear on how much inclusion is taking place in the process of economic growth. This holds true also for sustainability and governance. Sustainability concerns projects that directly deal with environmental protection as well as projects in other sectors which might have an environmental impact. An evaluation of one project, or of a specific sector, is not capable of assessing the impact on environmental protection. Likewise, governance is related to projects in various sectors.