Evaluating Climate Change Action for Sustainable Development

This chapter examines the demand for impact evidence and concludes that this demand goes beyond the experimental evidence that is produced during the lifetime of an intervention through “impact evaluations” as currently the term is used by many in the evidence movement. The demand for evidence of longer term impact at higher levels requires inspiration from an older tradition of impact evaluation and rethinking how the full range of impact evidence can be uncovered in evaluations. This is especially relevant for sustainable development which calls for a balanced approach on societal, economic and environmental issues. Climate change is a good example of this and a theory of change approach serves to identify key questions over time, space and scale to ensure that impact evidence can be found and reported throughout the lifetime of projects, programmes and policies and beyond in ex post impact assessments. Such an approach leads to mainstreaming of impact questions and related evaluation approaches throughout project and policy cycles. This chapter will demonstrate that evidence can be gathered throughout the lifetime of a project and beyond, in different geographic locations from very local to global, at different levels from relatively simple one dimensional interventions to multi-actor complex systems, up to global scales. It will thus argue for mainstreaming impact considerations throughout interventions, programmes and policies and for evaluations to gather evidence where it is available, rather than to focus the search for impact and its measurement on one or two causal mechanisms that are chosen for verification through experimentation.

The chapter presents a research for development program’s shift from a Logframe Approach to an outcome and results-based management oriented Monitoring, Evaluation and Learning (MEL) system. The CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) is designing an impact pathway-based MEL system that combines classic indicators of process in research with innovative indicators of change. We have developed a methodology for evaluating with our stakeholders factors that enable or inhibit progress towards behavioral outcomes in our sites and regions. Our impact pathways represent our best understanding of how engagement can bridge the gap between research outputs and outcomes in development. Our strategies for enabling change include a strong emphasis on partnerships, social learning, gender mainstreaming, capacity building, innovative communication and MEL that focuses on progress towards outcomes.

It presents the approach to theory of change, impact pathways and results-based management monitoring, evaluation and learning system. Our results highlight the importance of engaging users of our research in the development of Impact Pathways and continuously throughout the life of the program. Partnerships with diverse actors such as the private sector and policy makers is key to achieving change, like the attention to factors such as social learning, capacity building, networking and institutional change when generating evidence on climate smart technologies and practices. We conclude with insights on how the theory of change process in CGIAR can be used to achieve impacts that balance the drive to generate new knowledge in agricultural research with the priorities and urgency of the users and beneficiaries of these research results.

Evaluating the contribution of agricultural research to development has always been a challenge. Research alone does not lead to impact, but research does generate knowledge which actors, including development partners, can put into use to generate development outcomes. In CCAFS we are finding that a theory of change approach to research program design, implementation and evaluation is helping us bridge the gap between knowledge generation and development outcomes.

A stronger focus on results achieved in international cooperation on climate change has become common in the Swiss Agency for Development and Cooperation SDC (www.​eda.​admin.​ch/​sdc) and the State Secretariat for Economic Affairs SECO (www.​seco.​admin.​ch). In 2014 these agencies have commissioned an assessment on the effectiveness of more than 400 of their climate change interventions over the timeframe of 12 years (2000–2012). This paper presents the methodological approach of the assessment and its results. In a second step and most importantly, it summaries the challenges and lessons learnt of commissioning and conducting such a stock-taking exercise in the field of climate change. These lessons are addressed to evaluators, practitioners and policy makers. In general, the paper concludes that preparing such a report on the effectiveness of the international cooperation in climate change is indeed a very challenging exercise. More specifically, the paper argues that firstly many more efforts are needed from evaluators to identify best methodological practices in dealing with such a mass of information, the wide and highly diverse portfolio and a lack of good quantitative and qualitative data. Secondly, practitioners need to invest more in project design and in monitoring in order to provide accurate data as a basis for sound assessment. Finally, policy makers should be well aware of the significant investments needed for such assessments as an instrument of accountability. This paper thus contributes to the debate among interested stakeholders on the need for better results measurement and results reporting in international cooperation on climate change.

The Climate Change Sub-programme (CCSP) of the United Nations Environment Programme (UNEP) has four components: Adaptation, Mitigation, REDD+ and Science and Outreach. It cuts across all UNEP divisions located in Nairobi and Paris, and relies a lot on partnerships to drive its work and scale up its impact. The CCSP evaluation conducted by the UNEP Evaluation Office over the period 2013–2014, aimed at assessing the relevance and overall performance of the CCSP between 2008 and 2013. The complexity, geographical spread and rather weak results framework of the CCSP, coupled to rather limited evaluation resources and a shortage of evaluative evidence, required the Evaluation Office to develop an innovative analytical framework and data collection approach for this evaluation. It combined three areas of focus (strategic relevance, sub-programme performance and factors affecting performance), five interlinked units of analysis (UNEP corporate, sub-programme, country, component and project level), a Theory of Change approach and an appropriate combination of data collection tools. This chapter discusses the overall evaluation approach and process, followed by a summary of lessons learned which could be useful for future similar exercises.

This chapter looks at the initial progress of the JCM implementation in contributing to sustainable development in developing countries through facilitating diffusion of leading low-carbon technologies and implementation of mitigation actions. The current progress of the JCM in 16 partner countries looks promising with an established MRV system and efficient governance process. MRV methodologies are easy to use and benefits from standardized forms, default values, and practical monitoring system, but the methods in determining the reference emissions need to be strengthened. Rigorous project promotion is needed in underrepresented partner countries, especially least-developed countries, by supporting national programs and initiatives. The JCM should aim not only to complement, but also to improve preceding market mechanisms, by implementing a regulatory framework for evaluating its contributions to sustainable development. There is a need to clarify ways of credit allocation, arrange ways of credits accounting for national report and towards national pledge, and define the pathway of the JCM to a tradable crediting mechanism or retain its status quo of producing non-tradable credits.

Policies that target poverty reduction are often at odds with environmental sustainability. Assessing magnitudes of trade-offs between improved livelihoods on one side, and forest cover on the other, is important for designing win-win development policies that may help to mitigate climate change. I use a mix of panel data for 670 villages over a 10 year period, and combine it with historical land records and grey literature, to understand the drivers of deforestation within reserved forests of Thailand – an area where smallholder ethnic tribes are located. Given that reserved forests are the last bastions of forests in Thailand, examining what drives land clearing within these areas is important. I combine econometric findings with qualitative reports to infer that (i) it is important to measure the differential effects of policies on different crops, agricultural intensity and the agricultural frontier; and (ii) within forest reserves, policies that encourage cultivation overall may not be detrimental to forest cover after all. This has important implications for evaluators and policy makers.

This chapter presents the methodological approach adopted in the evaluation of GEF support to market change for climate change mitigation in four emerging markets: China, India, Mexico and Russia. The evaluation was completed in October 2013. This evaluation included 18 completed and fully evaluated GEF mitigation projects covering various sectors with opportunities for renewable energy, energy efficiency and methane emission reduction. A theory of change approach was used to undertake a comparative analysis across projects aiming to tease out changes across diverse markets or markets segments in different countries as a consequence of GEF support. While attention was given to the extent to which projects resulted in actual greenhouse gas (GHG) emission reductions, more emphasis was placed on understanding the extent and forms by which GEF projects contributed to long term market changes resulting in GHG emission reductions and assessing the added value of GEF support in the context of multiple factors affecting market change.

In many Least Developed Countries, the minimum level for basic services like energy access is not reached. In the cases of long-term investment in carbon intensive technologies, the expansion of basic services is likely to carry with it a significant increase in GHG emissions. This chapter discuss the importance of accounting for these avoided emissions through the case study of the Passive Solar Houses (PSH) in Afghanistan.

In Kabul winters are cold and 48 % of households cannot afford enough fuel to heat their house. To reduce fuels expenses and improve living conditions, the NGO GERES is supporting local artisans to disseminate a PSH model made of a veranda built on the south-facing part of the house to conserve the sun energy captured and stored in the walls. During the 2013–2014 winter, the fuel consumption and indoor temperature of PSH and control houses were monitored to assess the impact of the technology.

The results show an energy saving of 23 % resulting in annual greenhouse gases emission reduction of 0.37 tCO₂e/year as well as an average indoor temperature increase of 1.43 °C to reach 18.22 °C. Then, a regression model was developed to estimate the emissions that would have occurred if the control group had reached the same indoor temperature than the PSH and, in a second scenario, the minimum indoor temperature of 18 °C recommended by the WHO. For both scenario, the avoided emission represent approximately half of the total climate change mitigation impact with 0.40 tCO₂e/year and 0.34 tCO₂e/year respectively.

Decentralized renewable energy (DRE) projects have the potential to contribute to climate change mitigation, climate change adaptation, and sustainable development objectives. DRE systems are considered for emissions reduction or poverty alleviation purposes while their role for climate change adaptation has hardly been analysed. In terms of adaptation, DRE provides electricity that can be used both to prepare for and recover from disasters, and to provide additional income and livelihood opportunities, thus reducing dependency on natural resources. For example, DRE can power early warning systems, telecommunication systems, health clinics and potable water systems. Although it might be said that climate change adaptation applications of DRE systems have already been implemented, the vulnerability of these systems towards climate impacts, and the robustness of these systems to climatic impacts are oftentimes not even considered.

The assessment of 15 community-owned renewable energy projects in Guatemala and Nicaragua show that, under certain conditions, renewable energy projects can simultaneously meet the triple objective of sustainable development and climate change mitigation and adaptation. Research also points to specific drivers which can facilitate or hinder projects meeting their own stated objectives and, consequently, the triple objective, and their long-term functioning. These drivers include the specific background of the beneficiary community, the financing and implementing entities and the local governance structures in place.

In 2005, the international carbon market was launched under the Kyoto Protocol, creating an innovative financing design for low-emissions development initiatives. Just over 10 years after its inception, the carbon market can now provide insight on the opportunities and limitations of “blended finance” approaches, whereby private-public partnerships are employed to pursue global development goals such as poverty alleviation and development. Utilizing process-tracing and value chain methods, this chapter adds granularity to debates on whether and how carbon markets can support local economic development, as measured through the creation of local enterprises and the support of local livelihoods. It offers a “Livelihood Index” to assess the employment impact of the carbon intervention in order to address the core question: how is the carbon credit pie divvied up? Three carbon projects in Cambodia, aimed at household level interventions (water filters, biodigesters for cooking and fertilizer production, and fuel-efficient cookstoves) are evaluated through the livelihood index and results indicate that distribution strategies matter for local economic gains. Distribution strategies to deliver low-carbon technologies within the carbon market are currently a “black box”, understudied and undocumented in the project pipeline; this paper argues that opening the black box may be useful for policymakers, standard setting organizations and academics interested in promoting pro-poor impacts through carbon market interventions.

Evaluating climate change adaptation (CCA) interventions has yet proved to be a difficult task, as they involve a number of different stakeholders, time and geographical scale and political jurisdictions. As one effort to shed light on the subject, this paper presents the methodology and the results of a meta-analysis of ex-post evaluations of CCA programmes using a realist approach. This paper analyses CCA programmes in nine countries: Armenia, Egypt, Malawi, Mozambique, Namibia, the Philippines, Tanzania, Turkey and Zimbabwe. Together with their respective host governments, these programmes were implemented by either UNDP or various United Nations partner agencies and have already been evaluated by independent evaluators. Based on the analytical frameworks for evaluating CCA interventions, the authors hypothesized a number of key context, mechanism, and outcome configurations, which are considered vital in realist evaluation approach but have not yet been widely tested in the field of CCA. Although ex-post evaluations of multi-donor funded projects tend to be prepared out of bureaucratic requirement, the analytical method used in this paper, if used carefully, can unearth otherwise hidden important lessons and provide useful explanations. The results of the analysis can indicate that adopting a realist approach to complex development projects, such as these CCA programmes, is indeed a useful way of providing applicable explanations, rather than judgments, of what types of interventions may work for whom, how and in what circumstances for future CCA programming.

This chapter focuses on the evaluation of adaptive capacities of community-level human systems related to agriculture and food security. It highlights findings regarding approaches and domains to monitor and evaluate behavioral changes from CGIAR’s research program on climate change, agriculture and food security (CCAFS). This program, implemented in five West African countries, is intended to enhance adaptive capacities in agriculture management of natural resources and food systems. In support of participatory action research on climate-smart agriculture, a monitoring and evaluation plan was designed with the participation of all stakeholders to track changes in behavior of the participating community members. Individuals’ and groups’ stories of changes were collected using most significant change tools. The collected stories of changes were substantiated through field visits and triangulation techniques. Frequencies of the occurrence of characteristics of behavioral changes in the stories were estimated. The results show that smallholder farmers in the intervention areas adopted various characteristics of behavior change grouped into five domains: knowledge, practices, access to assets, partnership and organization. These characteristics can help efforts to construct quantitative indicators of climate change adaptation at local level. Further, the results suggest that application of behavioral change theories can facilitate the development of climate change adaptation indicators that are complementary to indicators of development outcomes. We conclude that collecting stories on behavioral changes can contribute to biophysical adaptation monitoring and evaluation.

This article highlights the process of using participatory approaches in measuring resilience using the Tracking Adaptation and Measuring Development (TAMD) Framework. The utilization of participatory approaches in Isiolo County using the TAMD framework is aligned to the recent thinking of measuring ‘subjective resilience’ using people’s perceptions to quantify household resilience. This article outlines the process of developing subjective indicators with communities, collection of baseline, monitoring and early outcome data by communities who were assisted in the development of their own adaptation theories of change. It also highlights the lessons and implications for policy if the approach is to be replicated at sub-national and community levels.

Whilst the principles of evaluating climate change adaptation are widely documented, there are many challenges in applying these principles in practice to evaluate, improve and learn from multi-sector, multi-scale and multi-stakeholder CCA initiatives with uncertain and future-oriented outcomes.

This chapter documents a research-evaluation approach applied during a 3-year, child-centred, community-based CCA project implemented in 40 barangays across four vulnerable provinces in the Philippines. The research aimed to help project implementers to learn from real-time feedback and perspectives from children and their communities and other participants. Researchers from the Institute for Sustainable Futures, University of Technology Sydney and practitioners from implementing NGOs Plan International and Save the Children collaborated on translating theory-based and development evaluation techniques into the field. We developed local-level indicators of adaptation, participatory focus group discussion and interview methods, and a guidance document for gathering and analysing evidence against these indicators.

Key to the success of this method was its participatory foundations – operationalising the principle that since ultimately adaptation is local, local voices and perspectives matter in understanding the impact of a project. Whilst there are limits to the “ideal” evaluation process, it is possible to achieve evaluative rigour in a process that is sensitive to the practical realities and pressures of project implementation. Embedding research and learning within practice – in the inherently uncertain context of supporting a community to adapt to climate change – provided new pathways for realising and sharing learnings to achieve better adaptation outcomes.

Brazil has historically coped with drought, a phenomenon that especially impacts the semi-arid lands of the Northeast. To deal with the various impacts of a current multi-year drought (2010-ongoing), the Government of Brazil, led by the Ministry of National Integration, partnered with the World Bank (WB) on a technical assistance program to foster proactive drought policy and management. The program works across sectors (climate/meteorology, water and sanitation, agriculture, environment, and disaster risk management) and levels (local, river-basin, urban, state, regional and federal) in relation to the outcomes and stakeholders it aims to engage and influence, and trough the integration of WB Global Practices and programs.

Inspired by successful models and lessons from other countries, the program aims to contribute to greater climate change resilience and reach a broad community of beneficiaries. To achieve these objectives, partners convened to (1) build a Northeast Drought Monitor; and (2) pilot drought preparedness plans across Northeast.

This chapter showcases the program and highlights key-milestones and direct and indirect outcomes identified by 2015. The institutional change process was assessed using qualitative analytical tools that integrate Outcome Mapping, the Capacity Development Results Framework, and Outcome Harvesting. Strengths, challenges, and outcomes (institutional changes) were identified, by tracking the program’s contribution throughout its duration and at its completion.

The evidence shows that the initiative was able to convene key-regional and federal level multi-sector stakeholders at a decisive moment, resulting in an unprecedented bottom-up and regionally-led collaboration. Through the engagement and commitment of the partners, the program fostered and coordinated continuous sharing of knowledge, data, and work between service providers, secretariats, municipalities and other stakeholders from distinct sectors and scales of decision making. Thus, it influenced progress towards overcoming some of the historical challenges related to drought management in Brazil.

With increasing implementation of climate change adaptation policies and projects as well as continued integration of adaptation into planning processes, there is an increasing need to understand the results of these adaptation interventions. Are they achieving their objectives? Are they actually leading to a reduction in vulnerability to climate change?

Monitoring and evaluation (M&E) can help answer these questions. However, due to the context specific and cross-sectoral nature of adaptation there is no one-size fits all approach to M&E. The Adaptation M&E Navigator helps to select a suitable M&E approach by providing a list of specific M&E purposes and matching them to relevant approaches. Key characteristics of each approach are highlighted to enable informed decision making. The Adaptation M&E Navigator also provides links to further guidance and examples from practice. The chapter outlines the rational and structure of the Adaptation M&E Navigator and how it can be used in practice.