13.1 Introduction
Climate change is a reality. Although it is important to acknowledge that the evidence of the linkage between rising economic loss of disasters and climate change has not been statistically established (Pielke 2014), changing precipitation and temperature patterns, as well as occasional hydro-meteorological extreme events, such as floods, droughts and landslides, have been hitting people especially at the community level, who have to rely on natural resources for their daily substance (Global Humanitarian Forum 2009). Reflecting the urgency and importance of climate change, the donor community for the past decade has been funding a number of climate change programmes in developing countries in close collaboration with host governments and various UN agencies. And it is in recent years that their initial implementation cycles have been completed and subsequently their ex-post evaluations have been conducted. In the meantime, discussions regarding evaluation practice, its criteria and framework specifically tailored to climate change projects and programmes have taken place, most notably through such communities of practice as Climate-Eval, the International Development Evaluation Association, and United Nations Evaluation Group.
Discussions in such arena have highlighted a number of difficulties related to evaluating climate change projects and programmes, including shifts in the objects of evaluation, new metrics, and greater focus on risk, uncertainty and complexity (Picciotto 2009). More specifically, evaluation of climate change adaptation (CCA) projects and programmes poses a number of difficulties and complications. For example, Valencia (2009) lists five types of such features: (1) “success” of CCA is when nothing happens; (2) evaluation of CCA occurs too early to tell whether the intervention has successfully withstood the projected impacts; (3) there are uncertainties of climate scenarios; (4) short-term weather variability disguises effectiveness of adaptation measures; and (5) contribution rather than attribution should be emphasized, because of the complexity of “overall adaptation process that is largely shaped by external factors” (Bours et al. 2014).
Advertisement
13.2 Approach and Study Material
In light of such background, the purpose of this paper is to adopt and test a certain philosophical lens, called critical realism, to a meta-analysis of CCA evaluation reports and to show implications of this approach for the current as well as future CCA programming.
The study material used was the evaluation reports of those CCA programmes that: (1) have been implemented by UNDP and other United Nations agencies; (2) have finished initial implementation cycles; and (3) have been subject to terminal evaluations. One of the unique aspects of these identified CCA programmes is that they represent the first evaluation results of the completed CCA programmes within the UNDP system (as of November 2014). Out of a total of 11, nine CCA programmes were selected based on the criterion that the quality of the evaluation reports was rated to be moderately satisfactory or higher by the UNDP Independent Evaluation Office.1 The authors conducted a meta-analysis of those ex-post evaluations by closely examining and comparing the contents of the evaluations by applying the philosophical lens of critical realism.
The nine programmes included were implemented in the following nine countries: Armenia, Egypt, Malawi, Mozambique, Namibia, the Philippines, Tanzania, Turkey and Zimbabwe (see Table 13.1 for summary). As the table shows, within the context of UN programming, these programmes vary in many aspects: the funding source (such as Global Environment Facility, Millennium Development Goals Achievement Fund, and United Nations internal resources); types of beneficiaries, target audiences and geographic regions (ranging from local vulnerable communities to inter-ministerial mainstreaming at the government level); and implementation modalities (including UNDP stand-alone, United Nations interagency joint programming and Delivering as One2).
Table 13.1
List of the CCA programme/project evaluation reports reviewed
Country | Programme/project title | Duration (months) | Implementation modality (funding source) |
|---|---|---|---|
Armenia | Adaptation to climate change impacts in mountain forest ecosystems of Armenia | May 2009 – Jun 2013 (50 m) | UNDP (GEF) |
Egypt | Joint programme: climate change risk management in Egypt | Oct 2008 – Apr 2013 (55 m) | JP (MDG-F) |
Malawi | The national programme for managing climate change in Malawi | Apr 2010 – Dec 2012 (33 m) | UNDP (AAP) |
Mozambique | Joint programme on environmental mainstreaming and adaptation to climate change in Mozambique | Sep 2008 – Aug 2012 (48 m) | JP (MDG-F) |
Namibia | Namibia country pilot partnership programme; adapting to climate change through the improvement of traditional crops and livestock farming | Jun 2007 – Dec 2011 (55 m) | UNDP (GEF) |
Philippines | Joint programme: strengthening the Philippines’ institutional capacity to adapt to climate change | Dec 2008 – Dec 2011 (37 m) | JP (MDG-F) |
Tanzania | Joint programme on environment with a focus on climate change, land degradation/desertification and natural resources management | Oct/Dec 2009 – Jun 2011 (21 m) | JP (MDG-F) |
Turkey | Joint programme on enhancing the xapacity of Turkey to adapt to climate change | Apr 2008 – Dec 2011 (45 m) | JP (MDG-F) |
Zimbabwe | Coping with drought and climate change in Zimbabwe project | May 2008 – Sep 2012 (53 m) | UNDP (GEF) |
Advertisement
This paper presents the findings of the meta-analysis conducted of the nine evaluation reports. Although the programmes evaluated vary from one another in many aspects, what is common is the structure of the evaluation reports. Each report consists of four major sections, each of which covers a specific evaluation criterion: relevance, efficiency, effectiveness and sustainability.
The evaluators who conducted the nine CCA programme evaluations all utilised the definitions of each criterion in Table 13.2, which are based on the OECD evaluation criteria adapted by UNDP and its partners (OECD 2002).3
Table 13.2
Definitions of evaluation criteria
Criteria | OECD definition |
|---|---|
Relevance | The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies |
Efficiency | A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results |
Effectiveness | The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance |
Sustainability
| The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time |
13.3 Realist Approach
This meta-analysis was conducted using a philosophical lens called critical realism. In evaluation, the realist approach emphasizes underlying assumptions about the way certain interventions are expected to yield certain outcomes in a certain context (Pawson and Tilley 2004). It thus defies the deterministic worldview which is symbolized as “if X happens, it automatically produces outcome Y.” Such a linear, sequential worldview is considered deterministic or positivistic, in that hypothesized theories of change are thought to work regardless of the context within which theories of change are situated. In other words, deterministic theory of change does not give us the explanations as to “for whom such interventions may work, in what circumstances, and how” (Pawson and Tilley 1997). Moreover, although the deterministic findings can tell us what interventions may have worked in certain countries under certain conditions (“there”), they may not tell us for whom these successful interventions are expected to work, under what circumstances, and how (“here”). The realist approach thus resonates with evidence-based policy making in that it is thought to be useful in answering the important evaluation question, i.e. “it worked there, but will it work here?” (Cartwright and Hardie 2012).
The following sections, however, first present the results of the meta-analysis that are considered deterministic in nature, immediately followed by non-deterministic ones and how the realist approach is applied. The intention behind this structure is to emphasize the characteristics of critical realism philosophy. Deterministic findings appear to help evaluators to know whether certain interventions work or not for achieving key outcomes, but such a deterministic approach is what a realist approach attempts to defy.
The realist approach belongs to the school of theory-based evaluation (Stern et al. 2012). The realist approach is based on a school of thought in a philosophy of science, called critical realism. The concept of critical realism has been most significantly developed by Roy Bhaskar.4 Critical realism can provide a useful lens especially in social sciences for the world that is “structured, differentiated, stratified and changing,” and recognizes the shift of emphasis “to what produces the events – not just to the events themselves.” (Danermark et al. 2002). An evaluation approach based on critical realism is thus an “intuitively appealing approach to those trying to expose and unpack the complexities of contexts and interrelated mechanisms underlying implementation activity” (Rycroft-Malone et al. 2012). The use of this evaluation approach is thus considered appropriate in the complex experience of CCA projects. Adoption of critical realism in evaluation field (principally in public health and criminology) has significantly progressed thanks to the work of Pawson (2013), Pawson et al. (2004), Pawson and Tilley (1997, 2004), and Wong et al. (2013) and other scholars.
However, a realist approach has not been widely conducted in international development, although some cases are found in a type of systematic reviews, e.g. Betts (2013). This meta-analysis is one such attempt. Quite unlike the conditions in making laboratory type experiments possible (“closed system”), critical realism acknowledges that the world is an “open system” consisting of things possessing causal powers (and also their potentialities) situated within many layers of structures (Bhaskar 2008). And because the world that people live in is an open system, it tells us that, unlike natural science, social science cannot predict things or present the world with successionist, cause-and-effects sequences.
The realist approach pays close attention to “contextual conditions” and how they influence mechanisms that generate (different) outcomes. It is a continuous, not a one-off, process of identifying specific contexts that may trigger some generative mechanisms to generate an outcome. Realist approach is thus about hypothesizing, selecting and refining so-called CMO (Context + Mechanism = Outcome) configurations.
13.4 Meta-analysis Conducted
The structure of the evaluations of the nine CCA programmes is based on the four evaluation criteria, i.e. relevance, efficiency, effectiveness and sustainability. Within this analysis framework, these criteria are considered as “outcomes” that lead to the ultimate CCA programme objectives. Within each outcome, there are several important intermediate outcomes (IOs) identified through the meta-analysis. Each IO is reported to have been influenced by a number of interventions on the ground.
According to Weiss (1997), a theory of change consists of two kinds of theories, i.e. implementation theory and programme theory. Implementation theory mainly pertains to programme activities or interventions themselves. It represents the assumptions that if certain interventions are implemented as planned, they are thought to generate desired results. Programme theory on the other hand represents the “ideas and assumptions [that] link the programme’s inputs to attainment of the desired ends” (Weiss 1997). It is not just what the programme activities are expected to achieve, but also how. The essence of such interventions and programme theories can be considered as a generative mechanism according to the realist approach and within CMO configurations.
The authors first extracted every single evaluative remark of these evaluations, each of which is categorized either ‘positive’ or ‘negative’. It altogether resulted in a total of 577 remarks gleaned out of the nine evaluations. Each of these remarks belonged to one or multiple evaluation criteria (i.e. relevance, efficiency, effectiveness, and sustainability). These remarks were then clustered according to: the evaluation criteria (i.e. outcomes5); intermediate outcomes (IOs) that lead to each evaluation criterion; and types of programme interventions implemented in achieving each IO. What this step enabled was a comparative analysis of the CCA programmes where similar interventions or activities across different CCA programmes were implemented. In other words, the meta-analysis conducted the following steps: identification and extraction of key IOs toward an outcome (each evaluation criterion); categorization of interventions to generate the corresponding IOs; development of hypothesis of programme theories that necessarily lead to an IO. And since this meta-analysis is based on the realist approach, it then sought contextual conditions that may or may not activate an underlying mechanism in generating IOs, and thus outcomes. It sought to identify theories of change for each outcome (evaluation criterion).
The following sections present first the M-O (mechanism = outcome) combinations for each criterion that can be estimated from analysing the CCA evaluations; and second, C (context) conditions which may or may not activate these M-O combinations, thereby showing a set of hypothesized CMO configurations. Each criterion is presented first only with M-O sequences, which represents a deterministic view. The latter half of the sections presents the contextual conditions, thereby completing the presentation of the hypothesized CMO configurations. Tables in the following sections present the summary of C-M-O configurations.
13.5 Mechanism-Outcome Sequences
13.5.1 Relevance M-O Sequences
Overall, a high degree of relevance is seen in all the studied CCA programmes. The joint programme for managing climate change risks in Egypt is found to be highly relevant in supporting Egypt to develop its climate change adaptation strategies. The programme in Mozambique is also found to be highly relevant to the national policy context, responding to the necessity to support institutional progress on CCA. Armenia’s programme focusing on its mountain forest ecosystem was evaluated to be well aligned with the national needs and priorities. Nonetheless, the aspect of relevance does not end with alignment at a national level. Tanzania’s programme has addressed problems of fuelwood availability and other means of improving livelihoods amongst local communities, reflecting the issues that had been considered high priority at a local level.
Through comparing the interventions taken place in each of the nine programmes from the point of view of the relevance criterion, the following theory of change was developed: “close coordination and working relationship with the national and local government enables both partners (government and United Nations implementing agency) to develop an appropriate CCA programme.” Here, the implementation theory part represents the type of similarly implemented interventions, and the programme theory part is a hypothesized mechanism of change attached to such implementation theory.
13.5.2 Efficiency M-O Sequences
Unlike relevance, for which it was relatively straightforward to construct a theory of change, all the other evaluation criteria were not necessarily straightforward, since each of the criteria can contain a number of different IOs to achieving a high level of an outcome. For the efficiency criterion, a number of IOs that helped achieve a high level of efficiency outcome were identified. The analysis was done by comparing similar interventions that were reported to have worked across the nine programmes.
As a result of a meta-analysis, stakeholder involvement at an early stage was identified as the first “recommended practice” to ensuring a high level of efficiency. In the Armenia, Mozambique and Zimbabwe programmes, there was active engagement of the stakeholders at a programme identification and planning stage. A corresponding hypothesis (i.e. programme theory) is that such an intervention activity fosters a high level of motivation and sense of ownership to the programme.
Four programmes, i.e. Egypt, Turkey, Armenia and Namibia, were reported to have achieved a high level of efficiency through strong financial controls, swift reporting, clarified roles and responsibilities and adaptive management through which the programmes were quick in responding to the changing needs and priorities of the beneficiaries on the ground. One way to achieving a high level of efficiency can thus be such interventions as adoption of adaptive management and clearly defined roles and responsibilities for involved parties. A corresponding programme theory can be that such adaptive management activities enable the programmes to attend to the needs and demands of the local beneficiaries whilst maintaining the ultimate programme goal.
13.5.3 Effectiveness M-O Sequences
The effectiveness criterion presents one of the most important aspects of programme’s success. Analysing the positive remarks found in the evaluation reports of the studied programmes has revealed that a high level of effectiveness is achieved, amongst others, through an IO of development of adaptive capacity and utilization of adaptive measures introduced by the programmes.
As a means to achieve such IO, training and transfer of techniques and practices for reducing the stakeholders’ vulnerability seemed to have ensured a high level of effectiveness of CCA programmes. Eight out of the nine programmes reported such activities and thus were evaluated positively for their effectiveness. For example, in Egypt, adaptive capacity was further enhanced within the Ministry of Agriculture and Land Reclamation in order for government staff to be able to forecast future scenarios in water and agriculture sectors. In Zimbabwe, a more accurate system of weather forecasts was introduced and capacity to manage the system was developed, thereby enabling high quality crop planting advice given to farmers. In Tanzania, the establishment of an environmental information system and a national environmental web portal were considered to be highly relevant adaptive measures that were introduced by the programme. The Namibia programme introduced such adaptive measures as dryland crop farming, conservation agriculture and improved seeds, and a drip irrigation system, all of which are reported to have played an important role in achieving a high level of effectiveness. A corresponding theory of change can thus be hypothesized as follows: “introduced adaptive measures and developed adaptive capacity facilitate these skills, techniques and knowledge to be kept applied and used.”
Realizing a wide range and level of mainstreaming is considered to be another IO in making a programme more effective. For example, in Turkey, a national climate change adaptation strategy and action plan was drafted and henceforth expected to be approved by a high level climate change coordination board. In Armenia, the introduced adaptive measures by the programme were successfully incorporated into an existing infrastructure that manages mountain forest ecosystems, including policy, legislation, institutions, procedures and mechanisms. In order to achieve such IO, provision of relevant technical, policy and advisory support to relevant stakeholders, from government staff to rural farmers have been reported to be effective. The corresponding programme theory here can be that provision of technical, policy and advisory support facilitates integration with “business-as-usual” infrastructures.
Another important IO that can lead to high effectiveness is a high level of awareness amongst the general public. Development and dissemination through documentary films, social network groups, large scale public events, TV and newspapers were seen in Egypt, Zimbabwe, the Philippines, Tanzania and Armenia. All these activities were reported to have contributed to realizing a high level of effectiveness by increasing awareness amongst the general public. One can thus infer that, in order to ensure a high level of effectiveness of a CCA programme, it is important to utilize various media, including face-to-face events, for wider publicity. A hypothesized programme theory here is that these events can attract attention and boost interest toward CCA amongst citizens.
13.5.4 Sustainability M-O Sequences
Since the studied evaluation reports were prepared right after the completion of programme activities, which corresponds to the second challenge discussed by Valencia (2009), it poses a significant challenge to evaluating the programme’s long-term sustainability. The meta-analysis nonetheless could identify some of the pertinent IOs and interventions, even if these were not explicitly identified in the evaluation reports.
The first IO for sustainability is “sustained built adaptive capacity, and a high utilization level of introduced adaptive measures.” Here an emphasis should be placed for sustaining (and not just one-off training of) the adaptive capacity that is built through programme activities, and a high level of utilization (and not just mere introduction) of adaptive measures. Hypothesized programme theory to ensuring them seems that such interventions foster a sense of ownership towards built capacities and introduced adaptive measures.
Sustained and high level of stakeholder engagement was identified as the second IO toward sustainability. The CCA programme in the Philippines has made sure that national and local partners continue similar activities and outputs that have been introduced by the programme. A hitherto non-existent network of environmental specialists was formed under the programme in Tanzania which since enabled all partners to work collaboratively.
The third IO identified was that mainstreaming at central policy and planning level is successful and sustained. The CCA programme of Tanzania has implemented its activities within the national institutional framework fully aligned with their national environmental policies. The programme also adopted a cross-cutting framework in order to mainstream environment and climate change issues into plans and policies of multiple sectors in the country. Similarly, in Mozambique, the programme has successfully integrated CCA activities in the country’s district-level strategic development and socioeconomic plan, the land use plan as well as integrated waste management plan. A theory of change, which is the combination of implementation theory and programme theory, can thus be hypothesized that CCA programme activities that are implemented within the local/national and existing institutional frameworks can foster a sense of ownership and trigger smooth integration in the target country’s planning and policies.
Fourth, high likelihood of generating broader adoption and replication is considered to be another IO that leads to a high level of sustainability. Introduction of adaptive measures to the stakeholders and institutions with relevant mandates seems to have yielded favourable results in achieving this positive IO. The programme activities in Egypt were well embedded into the work of the Agricultural Research Centre, whose relevant mandate successfully incorporated the new climate change risk research. A partnering technical university in Turkey is reported to be continuing to conduct a CCA related certification course which had been developed as part of the programme. A theory of change corresponding to this IO generation can be that the introduction of adaptive measures to the institutions already with relevant mandates can realize ‘rooting’ of such measures inside the institutions.
13.6 Contextual Conditions
Presented above was a series of M-O sequences without taking the contextual conditions into consideration. Such M-O only sequence, if used as it is, presents a deterministic view. Under such view, an underlying mechanism in generating above-mentioned IOs, namely the essence of programme theory, is believed to function everywhere, anytime, regardless of varying contexts. However, realist approach pays closer attention to the contextual conditions that necessarily allow such mechanism to function. In order to identify the contextual conditions, one needs first pay attention to those incidences where the identified theory of change did not work, i.e. those that have generated negative IOs. A general tendency amongst many meta-analyses of evaluation reports is to report what has worked in the effort to present so-called “best practices” by paying close attention to successful interventions and their programme theories. That approach risks missing lessons from failed interventions or strategies that may have worked only under specific conditions. The section below presents the findings about contextual conditions that have enabled (and not) a certain theory of change to work.
13.6.1 Context for Relevance
Almost all the evaluative remarks pertaining to the relevance criterion reported positive outcomes. But when focusing on those few incidences that were reported to have yielded slightly negative IOs, one can unearth the contextual conditions that may have helped this theory of change to trigger more successful IOs. In the case of Mozambique, even though there had been close coordination and working relationship with the national and local governments, relevance at a sub-national level was not considered high. In this case, local CCA priorities may not have been identified by the local governments and local partners. Similarly, in Turkey, because of abrupt insertion of carbon-footprint offsetting activities as part of CCA vulnerability reduction (though it is essentially for climate change mitigation
), the relevance level of this programme was not evaluated to be high.
From those incidences, one can hypothesize another contextual condition that may have allowed a theory of change (in this case in generating positive IOs for securing a high level of relevance) to work, i.e. that host government and line ministries have identified national and sectoral CCA priorities, or fully internalized the programme objectives specifically targeting adaptation. A set of identified CMO configurations for relevance criterion is shown in Table 13.3.
Table 13.3
Identified CMO configurations for relevance criterion
Context |
Theory of change
| Intermediate outcome | Outcome/criterion | ||
|---|---|---|---|---|---|
Host government and line ministries have already identified national and sectoral CCA priorities, and understand programme objective | + | Close coordination and working relationship with the national and local government enables both partners (government and United Nations implementing agency) to develop an appropriate CCA programme | = | High relevance of programme strategy and intervention components with national and global priorities | Relevance |
Local CCA priorities are identified by the local government and local partners |
13.6.2 Context for Efficiency
Referring to the estimated theory of change for realizing high stakeholder involvement, which is considered to be one of the key IOs in securing a high level of efficiency. Building partnerships at an early stage seems a common-sensical intervention to yield this IO. However, as reported in the case of Zimbabwe, even if partnerships are established at an early stage, when participating stakeholders are not well aware of CCA issues and risks and the CCA programme’s objectives, it is not likely for this corresponding theory of change to trigger a positive IO. Another contextual condition which can be identified for this theory of change from all of the studied evaluations is that the programme design is sector specific and focused rather than broad. Though this may not be a “recommended” context for a CCA programme because it can seem to be promoting a “silo” or sector-driven programme design, the degree of programme interventions’ focus seems to have enabled this theory of change to realize a high level of stakeholder involvement.
The second theory of change relates to another IO, i.e. level of programme management achievements. When a national programme management team (case of Tanzania) or national steering committee (case of Malawi) have not shown adequate leadership, the corresponding programme theory did not produce positive results. The more sector specific and focused the programme design is, the more positive patterns of results concerning this theory of change seem to be generated. Through a deterministic meta-analysis represented by mere M-O sequence, one could have ended the analysis in recommending adaptive management and clarified roles and responsibilities of the involved parties. The realistic approach can facilitate our thinking regarding the necessary contextual conditions and their hypotheses. A set of identified CMO configurations for efficiency criterion is shown in Table 13.4.
Table 13.4
Identified CMO configurations for efficiency criterion
Context |
Theory of change
| Intermediate outcome | Outcome/criterion | ||
|---|---|---|---|---|---|
Relevant stakeholders are supportive of United Nations and well aware of CCA issues and risks | + | Partnerships with stakeholders are built at an early stage, where they feel more motivated to participate in the programme | = | High stakeholder involvement | Efficiency |
Sector specific and focused programme design | |||||
Strong leadership from national executing agency | + | Adaptive management and clearly defined roles and responsibilities to each party enable the programme to attend to the needs and demands of the local beneficiaries whilst maintaining the ultimate programme goal
| = | High level of programme management achievements | |
Sector and region specific scope of programme |
13.6.3 Context for Effectiveness
There are three theories of change identified for the criterion of effectiveness. First one refers to the positive IO of a high level of adaptive capacity built and utilisation of adaptive measures. Contrary to the deterministic approach which automatically assumes the power of a mechanism (of generating an outcome) fully exercised regardless of context, realist approach pays close attention to the very structure wherein a mechanism is situated. For example, one contextual condition is the level of awareness of a local government partner. Local government partners can play a critical role in translating introduced adaptive measures and built adaptive capacity into actual benefits of the vulnerable people on the ground, such as rural farmers. If the introduced adaptive measures or built capacity is not clear to such partners, their utilization level can be quite limited. This refers to a case of Turkey where seasonal weather forecasts information provided over internet was introduced and related know-how taught. But since the end-users, e.g. rural farmers, were not reached, even though implementation theory may have held, the corresponding programme theory was not realized. Another contextual condition is where the types of adaptive capacity and adaptive measures are clear and well understood by those involved parties. In Namibia, the meteorological climate decision support tools were introduced to a government agency, but since the types of adaptive measures were not clear, introduced adaptive measures or built capacity did not generate a positive IO.
Second, in order for the theory of change for the wide range of mainstreaming to work, one can hypothesize, as part of the necessary contextual conditions, that relevant ministries and stakeholders should be highly aware of the climate risks and the vital importance of reducing vulnerability. A relevant contextual condition that is applicable for this theory of change is where government officials understand the actual need to integrate CCA issues in their business-as-usual activities. A case of Zimbabwe described the situation that, even though relevant and technical support was introduced, senior government officials did not fully appreciate the significance of such support, which thus did not yield a positive IO.
The third theory of change is about the raised level of awareness amongst the general public and government staff, since the level of awareness amongst them is considered key to achieving a high level of effectiveness. A relevant contextual condition for this theory of change that may alter the results of IO (i.e. high/low level of awareness) is that the general public is relatively unaware or lack knowledge of climate change risks. This condition should also be recognized as an important baseline situation under which planned interventions may trigger the corresponding programme theory in generating a positive IO. A set of identified CMO configurations for effectiveness criterion is shown in Table 13.5.
Table 13.5
Identified CMO configurations for effectiveness criterion
Context |
Theory of change
| Intermediate outcome | Outcome/criterion | ||
|---|---|---|---|---|---|
Specific types of skills that they need to acquire are clear to them | + | Training and transfer of techniques and practices for the relevant people facilitate these skills, techniques and knowledge to be applied and used | = | High level of adaptive capacity and utilisation of adaptive measures | Effectiveness |
Specifically identified types of participants are well aware of the climate risks | |||||
Relevant ministries and stakeholders are highly aware of the climate risks and the vital importance of reducing vulnerability | + | Provision of relevant technical, policy and advisory support to relevant people (from government staff to rural farmers) facilitates its integration with their “business-as-usual” activities | = | Wide range of mainstreaming | |
General citizens are relatively unaware or lack knowledge of climate change and associated risks | + | TV, newspaper and symposium for wider publicity attract attention and boost curiosity in citizens about CCA issues | = | Raised level of awareness amongst the general public |
13.6.4 Context for Sustainability
For this criterion, a high likelihood for sustaining built adaptive capacity and high utilisation level of adaptive measures introduced is considered to be one of the important IOs. In order for the corresponding theory of change for this IO to work, it is first necessary for the introduced adaptive capacities and measures to be those types that are needed and requested by end-users themselves (which was not the case in Mozambique). Sustained political interest towards the CCA programme’s intended objectives also need be present as another contextual condition that helps this theory of change to exercise its generative power.
Another IO that can contribute to achieving a high level of effectiveness is high likelihood for sustained, high level stakeholder engagement. One hypothesis for the contextual condition is where beneficiaries on the ground and government continue to be present and see the need and benefits in engaging themselves in the CCA programme’s intended objectives. This context can be hypothesized since there was one country case (Egypt) where the ultimate beneficiaries of the CCA programme, i.e. farmers, had not been in the programme activity process, which has negatively contributed to the sustainability element of this programme. Under such circumstances, though the corresponding implementation theory was held in all the programmes, the programme theory did not get to generate a positive outcome, if such contextual condition was not met.
The third IO in this criterion is about sustained level of mainstreaming at central policy and planning level. The corresponding implementation theory makes intuitive sense in that in order to mainstream CCA programme activities, they ought to be implemented within an existing local or national framework. However, in order for the corresponding programme theory to function and exercise its power, it seems to require a certain contextual condition where government counterparts understand the need of mainstreaming and a relatively high motivation level is found amongst government officials. One case (Zimbabwe) is reported to have designed and implemented a set of mainstreaming activities at central government level, but due to a lack of motivation of government counterparts, this theory of change did not see its generative power exercised.
The fourth IO pattern identified is about a high likelihood of generating broader adoption and replication in the long term. There are several cases identified through the meta-analysis where the corresponding theory of change did not generate such positive IO. The contextual conditions that can be extracted from these cases (Egypt and Mozambique) are that relevant stakeholders, such as government counterparts, have a strong sense of ownership, adequate resources and capabilities. Through the analysed cases, rooting of programme activities and intended directions within host government and agency seems well achieved under such contextual conditions. A set of identified CMO configurations for sustainability criterion is shown in Table 13.6.
Table 13.6
Identified CMO configurations for sustainability criterion
Context |
Theory of change
| Intermediate outcome | Outcome/criterion | ||
|---|---|---|---|---|---|
Key government counterparts, end-users and beneficiaries have relatively high levels of understanding of CCA programme’s intended objectives, and have clear ideas as to what types of adaptive capacity or measures they need | + |
Development of adaptive capacities and introduction of new adaptive measures that are requested by the end-users and can yield tangible results foster a sense of ownership towards built capacities and introduced measures | = | High likelihood for sustaining built adaptive capacity and high utilisation level of adaptive measures introduced | Sustainability |
Sustained political interest towards the CCA programme’s intended objectives | |||||
Beneficiaries on the ground and government continue to be present and see the need and benefits in engaging themselves to the CCA programme’s intended objectives | + | Formulation of communities of practice for developing and implementing new initiatives provides a useful platform for the committed partners/stakeholders to continue to be active for the CCA matters | = | High likelihood for sustained, high level stakeholder engagement | |
Government counterparts understand the need of mainstreaming | + | Programme activities implemented within the local/national and institutional existing framework foster a sense of ownership and trigger smooth integration of planning and policies | = | High likelihood for sustained level of mainstreaming at central policy and planning level | |
Institution’s sufficient resources and motivation level of government officials | |||||
Relevant stakeholders have strong sense of ownership and have adequate resources and capabilities | + | Introduction of adaptive measures to the stakeholders and institutions with relevant mandate enables ‘rooting’ of these measures inside the respective stakeholders and institutions | = | High likelihood of generating broader adoption and replications in the long term |
13.7 Methodological Implications
The purpose of this meta-analysis was to apply a critical realism philosophical lens and realist approach proposed by Pawson and Tilley (1997, 2004). Concretely, the purpose thus was to introduce and apply the method to extracting and hypothesising theories of change and contextual conditions under which programmes are expected to generate results through an underlying mechanism. In addition to a focus on programme theories, the realist approach pays close attention to the kinds of contextual conditions which enable (but not necessarily determine) a programme’s IOs to be realised. Therefore, the first implication of adopting a realist approach in a meta-analysis of CCA programmes is its focus on enabling contextual conditions. It can be a significant element since non-realistic evaluations often focus on the aspects that are only related to programme interventions and their programme theories and not such contexts.
Second, the contextual conditions that are identified and hypothesized in this meta-analysis can be useful for future CCA programming, particularly since similar types of interventions are often designed without necessarily thinking of the contexts. A realistic approach can provide explanations (rather than deterministic “answers”) as to what type of programme interventions may work under what type of conditions, and for whom. CCA programmes are embedded in quite a complex environment, e.g. involving a number of stakeholders and beneficiaries, implementing partners, funding sources and their requirements, and differing programme goals and local priorities, on top of the five types of challenges identified by Valencia (2009). All of these aspects can further be influenced by the country’s culture, history and socio-economic conditions. These are also important context aspects to explore in further deepening the CMO configurations for CCA programming. By paying close attention to such contextual conditions, the realist approach can thus be considered useful for knowing how, when and where to place the relevant interventions in a relevant context.
Third, this type of meta-analysis based on a realist approach may be able to shed new light onto a number of ex-post evaluations that have been already prepared. Though it will be difficult to prove quantitatively, there seems to be a tendency in the development practitioner’s community to pay inadequate attention to such ex-post evaluations, since they may be simply perceived as a mere requirement routinely asked by sponsoring agencies and donors. Since only in recent years have we started to complete ex-post evaluations of multilateral CCA programmes, a realist approach can provide a good analytical lens in fully utilizing those evaluations to better inform future CCA programming.
13.8 Conclusion
This paper presents a case of meta-analysis using a realist approach, the evaluation approach based on a philosophy of science called critical realism. The authors have adopted this approach in the meta-analysis of the nine CCA programme terminal evaluations, paying special attention to the context under which a mechanism is triggered to generate an IO. As a result, it could identify a number of pertinent programme theories and specific contextual conditions for each type of implemented interventions. This approach encourages the evaluator to go beyond deterministic cause-and-effect world and can provide explanations (rather than judgments) about what may work for whom, under what circumstances. CCA programmes by nature are quite complex, and are characterised by “multi-sectoral nature, cross-thematic focus, and long timeframes” (Bours et al. 2014), whilst impact of climate change felt differently in a different location and context. Thus simply collecting “best practices” of CCA interventions will not help policy makers and stakeholders to know what may work under their own circumstances, and how they are supposed to work for whom. What this analysis has revealed is that it is not just about “doing right things” or about “doing things right”; but it is also about “doing right things right, in right context”.
Some of the findings of this meta-analysis can indeed help provide useful explanations. For example, a rather usual intervention of closely coordinating with national and local government may not automatically produce the anticipated result of a higher level of relevance should the priorities of CCA not be identified by host government or line ministries prior to the programme. A result of an increased level of stakeholder involvement may not be guaranteed by simply building partnerships at an early stage; as it may depend on how specific and focused programme design is. Ensuring an increased level of adaptive capacity and a high level of utilisation of introduced adaptive measures is what virtually all CCA programmes wish to achieve through, e.g., facilitating training and transferring techniques and know-how. But even this may not work if specifically identified targeted groups of people are not well aware of climate risks, or cognisant of specific skills that they themselves want to acquire. Moreover, fostering a sense of ownership towards built capacities and introduced adaptive measures is key in generating the linkage between the programme’s inputs and attainment of the desired ends, in this case high likelihood of sustainability. But such generative mechanism may not be triggered under the context where key partners do not have a high level of understanding of programme’s intended overall objectives (as opposed to, e.g., their understanding toward introduced adaptive measures).
The CMO configurations presented in this paper should not, however, be considered a mere check-list for future CCA programming. Rather, they provide a good platform through which policy makers, programme designers and implementers can be guided, in order for them to make better decisions and develop CCA programmes that are suited for the respective circumstances.
Finally the authors would like to emphasize the point that adoption of realist approach in international development is still at its nascent stage. Exactly how critical realism should be adopted in international development evaluation still remains to be discussed and a challenge. Closer comparative examination of the framework put forward by Pawson (2013) and Wong et al. (2013), and its research implications in social sciences explained by Danermark et al. (2002) should be done to identify the methodological gaps (and potentially misapplied parts in our analysis), so that a realist approach can be more readily applied in evaluation of CCA and, more broadly, in international development evaluation.
Open Access This chapter is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the work’s Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work’s Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.